EventRefer: Attribute-Aware Grounding

• EventRefer is an attribute-aware grounding framework that localizes objects in dynamic scenes by processing voxelized event tensors, synchronized RGB frames, and structured referring expressions.
• It employs a DETR-style Transformer with attribute-aware masking and a Mixture of Event-Attribute Experts (MoEE) to dynamically fuse static, dynamic, egocentric, and relational cues.
• Empirical evaluations on the Talk2Event benchmark demonstrate improved accuracy across event-only, frame-only, and fusion settings, highlighting the benefits of its adaptive expert weighting.

Searching arXiv for the cited EventRefer and related Event Coreference papers. EventRefer is an attribute-aware grounding framework introduced with Talk2Event for language-driven object grounding in event-based perception. It operates on a voxelized event tensor, optionally a synchronized RGB frame, and a referring expression, and predicts a $2$D box $\hat{\mathbf b}=(x,y,w,h)$ localizing the described object. Within Talk2Event, the task is defined in three evaluation modes—event-only, frame-only, and event-frame fusion—and each referring expression is organized around four grounding attributes: Appearance $(\delta_{\mathbf a})$, Status $(\delta_{\mathbf s})$, Relation-to-Viewer $(\delta_{\mathbf v})$, and Relation-to-Others $(\delta_{\mathbf o})$ (Kong et al., 23 Jul 2025). A plausible source of terminological ambiguity is that the E$^3$C summary also uses “EventRefer” as a prospective system label for end-to-end event coreference; however, the concrete framework named EventRefer in the provided literature is the grounding model proposed with Talk2Event rather than the E$^3$C event coreference network (Lu et al., 2020, Kong et al., 23 Jul 2025).

1. Problem setting and representational assumptions

EventRefer is formulated for grounded understanding of dynamic scenes from event cameras. Its primary input is a voxelized event tensor

$$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$

where

$\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$

discretizes time into $\hat{\mathbf b}=(x,y,w,h)$0 bins (Kong et al., 23 Jul 2025). The model may also receive a synchronized RGB frame $\hat{\mathbf b}=(x,y,w,h)$1 at time $\hat{\mathbf b}=(x,y,w,h)$2, together with a referring expression $\hat{\mathbf b}=(x,y,w,h)$3. The prediction target is a $\hat{\mathbf b}=(x,y,w,h)$4D box $\hat{\mathbf b}=(x,y,w,h)$5 for the described object (Kong et al., 23 Jul 2025).

The linguistic side of the task is explicitly attribute-structured. Appearance denotes static cues such as color, shape, size, and class. Status denotes dynamic state, including moving, turning, and stopped. Relation-to-Viewer denotes egocentric position or distance, such as in front or to the left. Relation-to-Others denotes spatial or interaction context, such as next to a bus or behind a car (Kong et al., 23 Jul 2025). This decomposition matters because the benchmark and the model are both organized around the assumption that successful grounding in event-based perception requires simultaneous use of static, dynamic, egocentric, and relational cues.

This suggests that EventRefer is not merely a detector conditioned on text. Rather, it is a structured grounding system in which the language supervision is factorized into semantically distinct channels, and the visual side is expected to adapt to scene dynamics and modality availability.

2. Talk2Event benchmark and annotation protocol

Talk2Event is described as the first large-scale benchmark for language-driven object grounding in event-based perception, built from real-world driving data (Kong et al., 23 Jul 2025). The dataset contains $\hat{\mathbf b}=(x,y,w,h)$6 scenes, split into $\hat{\mathbf b}=(x,y,w,h)$7 train and $\hat{\mathbf b}=(x,y,w,h)$8 test scenes, with $\hat{\mathbf b}=(x,y,w,h)$9 unique objects and $(\delta_{\mathbf a})$0 validated referring expressions. The average expression length is approximately $(\delta_{\mathbf a})$1 words (Kong et al., 23 Jul 2025).

The data collection and annotation pipeline is explicitly staged. Frames are sampled at $(\delta_{\mathbf a})$2 Hz from the DSEC driving sequences. For each sampled frame, a $(\delta_{\mathbf a})$3 ms event volume $(\delta_{\mathbf a})$4 is extracted. Three captions per object are then generated via Qwen2-VL, guided to cover the four attributes by showing two RGB frames at $(\delta_{\mathbf a})$5 ms in a structured prompt labeled App., Status, Viewer, and Others. A second VLM prompt rewrites these descriptions for linguistic diversity. Attribute cue phrases are then fuzzy-matched into a binary token map $(\delta_{\mathbf a})$6, and human verification enforces uniqueness, attribute coverage of at least $(\delta_{\mathbf a})$7 of the $(\delta_{\mathbf a})$8 attributes, fluency, and grounding correctness (Kong et al., 23 Jul 2025).

Each caption consists of four short lines, one per attribute, followed by a summary sentence (Kong et al., 23 Jul 2025). This format is not incidental: it directly supports EventRefer’s positive-word matching and attribute-aware masking. Because the benchmark contains explicit token-span supervision for the four attribute channels, the model can learn not only which object to localize but also which parts of the sentence should dominate under different scene conditions.

3. Architecture: tokenization, fusion, and attribute-aware experts

EventRefer encodes $(\delta_{\mathbf a})$9, $(\delta_{\mathbf s})$0, and $(\delta_{\mathbf s})$1 into tokens, fuses them via a DETR-style Transformer, applies attribute-aware masking and a Mixture of Event-Attribute Experts (MoEE), and finally decodes into boxes and token alignments (Kong et al., 23 Jul 2025). The event encoder begins with a $(\delta_{\mathbf s})$2D convolutional stem on $(\delta_{\mathbf s})$3, downsamples spatially to $(\delta_{\mathbf s})$4, embeds into $(\delta_{\mathbf s})$5-D tokens, and then uses a $(\delta_{\mathbf s})$6-layer recurrent Transformer (RVT) with causal attention along time. When RGB frames are used, a ResNet-101 pretrained on ImageNet extracts multi-scale features from res3 to res5; each is projected to $(\delta_{\mathbf s})$7-D and flattened into tokens. The text encoder is RoBERTa-base, which maps $(\delta_{\mathbf s})$8 into $(\delta_{\mathbf s})$9-D vectors that are then linearly projected to $(\delta_{\mathbf v})$0-D (Kong et al., 23 Jul 2025).

For multimodal fusion, all tokens are concatenated and passed through a shared $(\delta_{\mathbf v})$1-layer Transformer encoder. A query set of $(\delta_{\mathbf v})$2 learnable DETR queries attends to the fused tokens, producing hidden states $(\delta_{\mathbf v})$3 (Kong et al., 23 Jul 2025). This provides the common query representation on which the attribute-aware stages operate.

Positive-word matching is defined separately for each attribute $(\delta_{\mathbf v})$4. Cue phrases and synonyms in the raw caption are matched to locate token spans, yielding the binary map $(\delta_{\mathbf v})$5. The map is normalized by softmax over positive tokens to form a token-attention prior. A supplementary map $(\delta_{\mathbf v})$6 is defined for tokens not assigned to any specific attribute and serves as public context (Kong et al., 23 Jul 2025).

The attribute-aware representation stage combines $(\delta_{\mathbf v})$7 with $(\delta_{\mathbf v})$8, forming a mask $(\delta_{\mathbf v})$9, and applies element-wise masking to $(\delta_{\mathbf o})$0 to produce $(\delta_{\mathbf o})$1. Each masked representation is passed through a small FFN, yielding an expert feature $(\delta_{\mathbf o})$2 (Kong et al., 23 Jul 2025). The effect is to instantiate four expert streams, each centered on one grounding attribute while retaining access to public context.

4. Mixture of Event-Attribute Experts and dynamic weighting

The core fusion mechanism in EventRefer is the Mixture of Event-Attribute Experts. For each attribute-specific expert, the hidden states are mean-pooled over queries: $(\delta_{\mathbf o})$3 The four pooled experts are concatenated and projected to four gating logits through $(\delta_{\mathbf o})$4, and Gaussian noise $(\delta_{\mathbf o})$5, scaled by learnable $(\delta_{\mathbf o})$6, is added before softmax: $(\delta_{\mathbf o})$7 The final fused representation is

$(\delta_{\mathbf o})$8

The interpretation given in the source is that $(\delta_{\mathbf o})$9 adaptively emphasizes the most informative attribute, for example Status in high-motion scenes and Appearance in static daylight (Kong et al., 23 Jul 2025).

This adaptive mechanism is one of the defining properties of EventRefer. It does not assume that every referring expression or every sensory stream should weight the four attributes equally. Instead, the model learns a scene- and modality-dependent mixture over attribute experts. The reported expert-activation analyses are consistent with this design. In the event-only setting, small dynamic classes such as Rider and Bike show high Status $^3$0 activation, while large static classes such as Bus and Truck favor Appearance $^3$1 or Viewer $^3$2. Across response-strength bins, low event density is dominated by $^3$3 and $^3$4, whereas high event density shifts toward $^3$5 and $^3$6, capturing motion and inter-object relations (Kong et al., 23 Jul 2025).

A plausible implication is that MoEE functions as a conditional routing mechanism over semantically typed grounding evidence rather than a generic feature blender.

5. Objectives, matching, and inference

Training is cast as multi-attribute grounding via multi-object matching. Each ground-truth box $^3$7 is duplicated four times, once per attribute (Kong et al., 23 Jul 2025). Hungarian matching is then performed between query $^3$8 and target $^3$9 using the cost

$^3$0

where $^3$1 and $^3$2 are the box and token logits predicted by query $^3$3 (Kong et al., 23 Jul 2025). Box regression uses $^3$4 and Generalized IoU losses, while the attribute alignment loss $^3$5 is a cross-entropy between softmaxed $^3$6 and ground-truth $^3$7. The total loss is the sum of $^3$8 over matched pairs (Kong et al., 23 Jul 2025).

At inference time, each query emits a box $^3$9 and token logits $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$0. For each attribute $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$1, a token map $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$2 is constructed through positive-word matching, and a score is computed by

$$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$3

The final prediction is the box $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$4 with highest $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$5 (Kong et al., 23 Jul 2025). The same network can operate with only event tokens, only frame tokens, or both concatenated, and MoEE is reported to shift gating automatically based on available cues (Kong et al., 23 Jul 2025).

This training-and-inference design tightly couples localization and token-level attribute alignment. The grounding decision is therefore mediated both by geometric fit and by consistency between predicted token distributions and attribute-specific token priors.

6. Empirical performance and ablation findings

The reported overall results use [email protected] and Top-1 Acc., with mAcc and mIoU expressed in percent (Kong et al., 23 Jul 2025). In the frame-only setting, EventRefer achieves $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$6, compared with the best baseline BUTD-DETR at $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$7. In the event-only setting, EventRefer obtains $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$8, compared with EvRT-DETR at $$\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),$$9. In event-frame fusion, EventRefer reaches $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$0, compared with FlexEvent at $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$1 (Kong et al., 23 Jul 2025).

The event-only ablations identify the contribution of each major component. A vanilla configuration without PWM, MAF, or MoEE achieves $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$2 mAcc. Adding PWM only yields $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$3 $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$4, MAF only yields $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$5 $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$6, PWM+MAF yields $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$7 $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$8, and PWM+MAF+MoEE yields $\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor$9 $\hat{\mathbf b}=(x,y,w,h)$00 (Kong et al., 23 Jul 2025). Single-attribute experiments show $\hat{\mathbf b}=(x,y,w,h)$01 for Appearance only, $\hat{\mathbf b}=(x,y,w,h)$02 for Status only, $\hat{\mathbf b}=(x,y,w,h)$03 for Viewer only, and $\hat{\mathbf b}=(x,y,w,h)$04 for Others only, while using all four through MoEE reaches $\hat{\mathbf b}=(x,y,w,h)$05, a $\hat{\mathbf b}=(x,y,w,h)$06 gain over the best single attribute (Kong et al., 23 Jul 2025).

The comparison of fusion strategies further isolates the value of MoEE. No fusion, implemented as feature summation, achieves $\hat{\mathbf b}=(x,y,w,h)$07 mAcc; additive fusion reaches $\hat{\mathbf b}=(x,y,w,h)$08; concatenation reaches $\hat{\mathbf b}=(x,y,w,h)$09; learned attention reaches $\hat{\mathbf b}=(x,y,w,h)$10; and MoEE reaches $\hat{\mathbf b}=(x,y,w,h)$11, a $\hat{\mathbf b}=(x,y,w,h)$12 improvement over the no-fusion baseline (Kong et al., 23 Jul 2025). These results support the claim that the framework’s gains are not reducible to multimodal fusion alone, but depend on attribute-aware decomposition and dynamic expert weighting.

7. Relation to event coreference and broader interpretation

The E$\hat{\mathbf b}=(x,y,w,h)$13C paper, “End-to-End Neural Event Coreference Resolution,” addresses a different problem: joint event detection and event coreference resolution from raw text (Lu et al., 2020). It uses a multi-layer scalar mixture of BERT embeddings, masked self-attention with window size $\hat{\mathbf b}=(x,y,w,h)$14, a mention proposal network restricted to single tokens, and an antecedent prediction network in which event detection is integrated by assigning non-event spans to a dummy antecedent $\hat{\mathbf b}=(x,y,w,h)$15. It further introduces a type-guided event coreference mechanism with type prediction, type-refined mention representations, and type-guided decoding, and reports new state-of-the-art performance on KBP 2016 and KBP 2017 (Lu et al., 2020).

The E$\hat{\mathbf b}=(x,y,w,h)$16C summary explicitly presents its design as a blueprint for an “EventRefer” system and recommends a fully end-to-end architecture, a strong pre-trained encoder with masked local/self-attention, integration of event typing as an auxiliary but critical subtask, and joint optimization over mention and antecedent losses (Lu et al., 2020). However, those recommendations concern event coreference in text rather than language-driven object grounding from event cameras. The overlap lies mainly at the level of system philosophy: both frameworks reject brittle pipeline decomposition, both rely on learned representations rather than hand-crafted features, and both inject structured semantic guidance into end-to-end inference (Lu et al., 2020, Kong et al., 23 Jul 2025).

This suggests that “EventRefer” names two distinct research directions in the supplied material. In one sense, it denotes a prospective event coreference system inspired by E$\hat{\mathbf b}=(x,y,w,h)$17C; in the established sense introduced by Talk2Event, it denotes an attribute-aware grounding framework for dynamic visual scenes captured by event cameras. The latter usage is the one tied to a specific architecture, benchmark, and experimental record (Kong et al., 23 Jul 2025).