---
title: 'EventRefer: Attribute-Aware Grounding'
url: https://www.emergentmind.com/topics/eventrefer
type: topic
---

# EventRefer: Attribute-Aware Grounding

Searching arXiv for the cited EventRefer and related Event Coreference papers.
EventRefer is an attribute-aware grounding framework introduced with Talk2Event for language-driven object grounding in event-based perception. It operates on a voxelized event tensor, optionally a synchronized RGB frame, and a referring expression, and predicts a \(2\)D box \(\hat{\mathbf b}=(x,y,w,h)\) localizing the described object. Within Talk2Event, the task is defined in three evaluation modes—event-only, frame-only, and event-frame fusion—and each referring expression is organized around four grounding attributes: Appearance \((\delta_{\mathbf a})\), Status \((\delta_{\mathbf s})\), Relation-to-Viewer \((\delta_{\mathbf v})\), and Relation-to-Others \((\delta_{\mathbf o})\) [2507.17664]. A plausible source of terminological ambiguity is that the E\(^3\)C summary also uses “EventRefer” as a prospective system label for end-to-end event coreference; however, the concrete framework named EventRefer in the provided literature is the grounding model proposed with Talk2Event rather than the E\(^3\)C event coreference network [2009.08153][2507.17664].

## 1. Problem setting and representational assumptions

EventRefer is formulated for grounded understanding of dynamic scenes from event cameras. Its primary input is a voxelized event tensor
\[
\mathbf{E}(p,\tau,x,y) \;=\;\sum_{e_k\in\mathcal{E}} \delta(p-p_k)\,\delta(x-x_k)\,\delta(y-y_k)\,\delta\bigl(\tau-\tau_k\bigr),
\]
where
\[
\tau_k=\lfloor(t_k-t_a)/(t_b-t_a)\times T\rfloor
\]
discretizes time into \(T\) bins [2507.17664]. The model may also receive a synchronized RGB frame \(\mathbf{F}\in\mathbb R^{3\times H\times W}\) at time \(t_0\), together with a referring expression \(\mathcal{S}=\{w_1,\dots,w_C\}\). The prediction target is a \(2\)D box \(\hat{\mathbf b}=(x,y,w,h)\) for the described object [2507.17664].

The linguistic side of the task is explicitly attribute-structured. Appearance denotes static cues such as color, shape, size, and class. Status denotes dynamic state, including moving, turning, and stopped. Relation-to-Viewer denotes egocentric position or distance, such as in front or to the left. Relation-to-Others denotes spatial or interaction context, such as next to a bus or behind a car [2507.17664]. This decomposition matters because the benchmark and the model are both organized around the assumption that successful grounding in event-based perception requires simultaneous use of static, dynamic, egocentric, and relational cues.

This suggests that EventRefer is not merely a detector conditioned on text. Rather, it is a structured grounding system in which the language supervision is factorized into semantically distinct channels, and the visual side is expected to adapt to scene dynamics and modality availability.

## 2. Talk2Event benchmark and annotation protocol

Talk2Event is described as the first large-scale benchmark for language-driven object grounding in event-based perception, built from real-world driving data [2507.17664]. The dataset contains \(5\,567\) scenes, split into \(4\,433\) train and \(1\,134\) test scenes, with \(13\,458\) unique objects and \(30\,690\) validated referring expressions. The average expression length is approximately \(34.1\) words [2507.17664].

The data collection and annotation pipeline is explicitly staged. Frames are sampled at \(5\) Hz from the DSEC driving sequences. For each sampled frame, a \(200\) ms event volume \([t_0-100\text{ms},\,t_0+100\text{ms}]\) is extracted. Three captions per object are then generated via Qwen2-VL, guided to cover the four attributes by showing two RGB frames at \(t_0\pm200\) ms in a structured prompt labeled App., Status, Viewer, and Others. A second VLM prompt rewrites these descriptions for linguistic diversity. Attribute cue phrases are then fuzzy-matched into a binary token map \(\mathbf m_i\in\{0,1\}^C\), and human verification enforces uniqueness, attribute coverage of at least \(2\) of the \(4\) attributes, fluency, and grounding correctness [2507.17664].

Each caption consists of four short lines, one per attribute, followed by a summary sentence [2507.17664]. This format is not incidental: it directly supports EventRefer’s positive-word matching and attribute-aware masking. Because the benchmark contains explicit token-span supervision for the four attribute channels, the model can learn not only which object to localize but also which parts of the sentence should dominate under different scene conditions.

## 3. Architecture: tokenization, fusion, and attribute-aware experts

EventRefer encodes \(\mathbf E\), \(\mathbf F\), and \(\mathcal S\) into tokens, fuses them via a DETR-style Transformer, applies attribute-aware masking and a Mixture of Event-Attribute Experts (MoEE), and finally decodes into boxes and token alignments [2507.17664]. The event encoder begins with a \(3\)D convolutional stem on \(\mathbf E\in\mathbb R^{2\times T\times H\times W}\), downsamples spatially to \(H'\times W'\), embeds into \(256\)-D tokens, and then uses a \(6\)-layer recurrent Transformer (RVT) with causal attention along time. When RGB frames are used, a ResNet-101 pretrained on ImageNet extracts multi-scale features from res3 to res5; each is projected to \(256\)-D and flattened into tokens. The text encoder is RoBERTa-base, which maps \(\{w_j\}_{j=1}^C\) into \(768\)-D vectors that are then linearly projected to \(256\)-D [2507.17664].

For multimodal fusion, all tokens are concatenated and passed through a shared \(6\)-layer Transformer encoder. A query set of \(100\) learnable DETR queries attends to the fused tokens, producing hidden states \(\mathbf H\in\mathbb R^{B\times Q\times C}\) [2507.17664]. This provides the common query representation on which the attribute-aware stages operate.

Positive-word matching is defined separately for each attribute \(\delta_i\in\{\mathbf a,\mathbf s,\mathbf v,\mathbf o\}\). Cue phrases and synonyms in the raw caption are matched to locate token spans, yielding the binary map \(\mathbf m_i\in\{0,1\}^C\). The map is normalized by softmax over positive tokens to form a token-attention prior. A supplementary map \(\mathbf m_0\) is defined for tokens not assigned to any specific attribute and serves as public context [2507.17664].

The attribute-aware representation stage combines \(\mathbf m_i\) with \(\mathbf m_0\), forming a mask \(\mathbf m_i^{\text{att}}=\mathbf m_i\lor\mathbf m_0\), and applies element-wise masking to \(\mathbf H\) to produce \(\mathbf H_i^{\text{att}}\in\mathbb R^{B\times Q\times C}\). Each masked representation is passed through a small FFN, yielding an expert feature \(\mathbf H_i^{\mathrm{exp}}\) [2507.17664]. The effect is to instantiate four expert streams, each centered on one grounding attribute while retaining access to public context.

## 4. Mixture of Event-Attribute Experts and dynamic weighting

The core fusion mechanism in EventRefer is the Mixture of Event-Attribute Experts. For each attribute-specific expert, the hidden states are mean-pooled over queries:
\[
\bar{\mathbf h}_i=\tfrac1Q\sum_{q=1}^Q\mathbf H_i^{\mathrm{exp}}[q]\in\mathbb R^{B\times C}.
\]
The four pooled experts are concatenated and projected to four gating logits through \(\mathbf W\in\mathbb R^{4\times C}\), and Gaussian noise \(\epsilon\sim\mathcal N(0,1)\), scaled by learnable \(\sigma\), is added before softmax:
\[
\boldsymbol\lambda = \mathrm{softmax}\bigl([\bar h_1;\dots;\bar h_4]\,\mathbf W \;+\;\sigma\,\epsilon\bigr),\quad \lambda_i\ge0,\;\sum_i\lambda_i=1.
\]
The final fused representation is
\[
\mathbf H^{\mathrm{fuse}} \;=\;\sum_{i=1}^{4}\lambda_i\,\mathbf H_i^{\mathrm{exp}}\in\mathbb R^{B\times Q\times C}.
\]
The interpretation given in the source is that \(\lambda_i\) adaptively emphasizes the most informative attribute, for example Status in high-motion scenes and Appearance in static daylight [2507.17664].

This adaptive mechanism is one of the defining properties of EventRefer. It does not assume that every referring expression or every sensory stream should weight the four attributes equally. Instead, the model learns a scene- and modality-dependent mixture over attribute experts. The reported expert-activation analyses are consistent with this design. In the event-only setting, small dynamic classes such as Rider and Bike show high Status \((\delta_{\mathbf s})\) activation, while large static classes such as Bus and Truck favor Appearance \((\delta_{\mathbf a})\) or Viewer \((\delta_{\mathbf v})\). Across response-strength bins, low event density is dominated by \(\delta_{\mathbf a}\) and \(\delta_{\mathbf v}\), whereas high event density shifts toward \(\delta_{\mathbf s}\) and \(\delta_{\mathbf o}\), capturing motion and inter-object relations [2507.17664].

A plausible implication is that MoEE functions as a conditional routing mechanism over semantically typed grounding evidence rather than a generic feature blender.

## 5. Objectives, matching, and inference

Training is cast as multi-attribute grounding via multi-object matching. Each ground-truth box \(\mathbf b\) is duplicated four times, once per attribute [2507.17664]. Hungarian matching is then performed between query \(n\) and target \(i\) using the cost
\[
\mathcal C_{(n,i)}
=\beta_{\rm box}\Bigl(\|\hat{\mathbf b}_n-\mathbf b_i\|_1 + \mathrm{GIoU}(\hat{\mathbf b}_n,\mathbf b_i)\Bigr)
+\beta_{\rm attr}\,\mathcal L_{\rm attr}\!\bigl(\hat{\mathbf m}_n,\mathbf m_i\bigr),
\]
where \(\hat{\mathbf b}_n\) and \(\hat{\mathbf m}_n\) are the box and token logits predicted by query \(n\) [2507.17664]. Box regression uses \(\ell_1\) and Generalized IoU losses, while the attribute alignment loss \(\mathcal L_{\rm attr}\) is a cross-entropy between softmaxed \(\hat{\mathbf m}_n\) and ground-truth \(\mathbf m_i\). The total loss is the sum of \(\mathcal C_{(n,i)}\) over matched pairs [2507.17664].

At inference time, each query emits a box \(\hat{\mathbf b}_n\) and token logits \(\hat{\mathbf m}_n\). For each attribute \(i\), a token map \(\mathbf m_i\) is constructed through positive-word matching, and a score is computed by
\[
\text{score}_{(n,i)}
=\bigl\langle\mathrm{softmax}(\hat{\mathbf m}_n),\,\mathrm{softmax}(\mathbf m_i)\bigr\rangle.
\]
The final prediction is the box \(\hat{\mathbf b}_n\) with highest \(\max_i\text{score}_{(n,i)}\) [2507.17664]. The same network can operate with only event tokens, only frame tokens, or both concatenated, and MoEE is reported to shift gating automatically based on available cues [2507.17664].

This training-and-inference design tightly couples localization and token-level attribute alignment. The grounding decision is therefore mediated both by geometric fit and by consistency between predicted token distributions and attribute-specific token priors.

## 6. Empirical performance and ablation findings

The reported overall results use IoU@0.95 and Top-1 Acc., with mAcc and mIoU expressed in percent [2507.17664]. In the frame-only setting, EventRefer achieves \(55.47/85.76\), compared with the best baseline BUTD-DETR at \(48.91/84.30\). In the event-only setting, EventRefer obtains \(31.96/76.46\), compared with EvRT-DETR at \(29.34/75.66\). In event-frame fusion, EventRefer reaches \(61.82/87.32\), compared with FlexEvent at \(59.40/86.83\) [2507.17664].

The event-only ablations identify the contribution of each major component. A vanilla configuration without PWM, MAF, or MoEE achieves \(22.07\) mAcc. Adding PWM only yields \(26.38\) \((+4.31)\), MAF only yields \(27.01\) \((+4.94)\), PWM+MAF yields \(29.66\) \((+7.59)\), and PWM+MAF+MoEE yields \(31.96\) \((+9.89)\) [2507.17664]. Single-attribute experiments show \(27.98\) for Appearance only, \(28.90\) for Status only, \(27.03\) for Viewer only, and \(26.97\) for Others only, while using all four through MoEE reaches \(31.96\), a \(+3.98\) gain over the best single attribute [2507.17664].

The comparison of fusion strategies further isolates the value of MoEE. No fusion, implemented as feature summation, achieves \(26.38\) mAcc; additive fusion reaches \(28.39\); concatenation reaches \(27.50\); learned attention reaches \(29.66\); and MoEE reaches \(31.96\), a \(+5.58\) improvement over the no-fusion baseline [2507.17664]. These results support the claim that the framework’s gains are not reducible to multimodal fusion alone, but depend on attribute-aware decomposition and dynamic expert weighting.

## 7. Relation to event coreference and broader interpretation

The E\(^3\)C paper, “End-to-End Neural Event Coreference Resolution,” addresses a different problem: joint event detection and event coreference resolution from raw text [2009.08153]. It uses a multi-layer scalar mixture of BERT embeddings, masked self-attention with window size \(c=10\), a mention proposal network restricted to single tokens, and an antecedent prediction network in which event detection is integrated by assigning non-event spans to a dummy antecedent \(\epsilon\). It further introduces a type-guided event coreference mechanism with type prediction, type-refined mention representations, and type-guided decoding, and reports new state-of-the-art performance on KBP 2016 and KBP 2017 [2009.08153].

The E\(^3\)C summary explicitly presents its design as a blueprint for an “EventRefer” system and recommends a fully end-to-end architecture, a strong pre-trained encoder with masked local/self-attention, integration of event typing as an auxiliary but critical subtask, and joint optimization over mention and antecedent losses [2009.08153]. However, those recommendations concern event coreference in text rather than language-driven object grounding from event cameras. The overlap lies mainly at the level of system philosophy: both frameworks reject brittle pipeline decomposition, both rely on learned representations rather than hand-crafted features, and both inject structured semantic guidance into end-to-end inference [2009.08153][2507.17664].

This suggests that “EventRefer” names two distinct research directions in the supplied material. In one sense, it denotes a prospective event coreference system inspired by E\(^3\)C; in the established sense introduced by Talk2Event, it denotes an attribute-aware grounding framework for dynamic visual scenes captured by event cameras. The latter usage is the one tied to a specific architecture, benchmark, and experimental record [2507.17664].

Source: https://www.emergentmind.com/topics/eventrefer