Mixture of Event-Attribute Experts (MoEE)

• MoEE is a dynamic fusion mechanism that employs four specialized experts (appearance, status, egocentric, inter-object) for attribute-aware visual grounding.
• It uses a dedicated gating network with stochastic regularization to adaptively weight expert outputs, improving alignment between visual and linguistic cues.
• Empirical results demonstrate significant performance gains over traditional fusion methods in event-only, frame-only, and joint scenarios.

The Mixture of Event-Attribute Experts (MoEE) is a dynamic fusion mechanism for attribute-aware visual grounding, specifically developed for event-based perception tasks that require multimodal and temporally-aware reasoning. MoEE equips the EventRefer framework with the ability to adaptively select and synthesize cues derived from four distinct attribute “experts”—appearance, status (motion/state), relation-to-viewer (egocentric), and relation-to-others (inter-object)—to ground natural language referring expressions in asynchronous visual streams. This operational principle has enabled EventRefer to achieve consistent performance gains in both event-only, frame-only, and joint event-frame scenarios, marking a significant advance in grounded scene understanding from event cameras (Kong et al., 23 Jul 2025).

1. High-Level Architecture

MoEE instantiates four parallel attribute-specific experts, each corresponding to a major axis of linguistic and perceptual grounding:

• Appearance (δₐ): Encodes visual appearance information.
• Status (δₛ): Captures motion or instantaneous state.
• Relation-to-Viewer (δᵥ): Models egocentric spatial and semantic relationships.
• Relation-to-Others (δₒ): Encodes inter-object contexts.

The workflow can be summarized as follows:

1. Input Encoding: Event frames $E$ and optional RGB frames $F$ are encoded via a dedicated event transformer and a ResNet-101 backbone, respectively. A shared RoBERTa text encoder (max 64 tokens, projected from 768 to 256 dimensions) provides the linguistic embeddings.
2. Fusion Backbone: Multimodal tokens are concatenated and further encoded using a DETR-style Transformer, producing hidden states $H \in \mathbb{R}^{B \times Q \times C}$, with $B$ the batch size, $Q=100$ queries, and $C\sim256$.
3. Attribute Masking: For each attribute index $i\in\{\text{a},\text{s},\text{v},\text{o}\}$, H is masked using attribute-specific cues $m_i$ (derived via fuzzy phrase matching in the referring expression) and a public context mask $m_0$, producing $H^{\text{att}}_i$.
4. Expert Feature Extraction: Each $F$0 passes through a distinct lightweight feedforward network (FFN) to yield expert feature maps $F$1.
5. Gating and Mixture: A gating network receives mean-pooled summaries of each expert. Mixture weights $F$2 are computed by linearly projecting the concatenated summaries, adding trainable Gaussian noise, and applying a softmax.
6. Dynamic Fusion: The weighted sum of expert outputs $F$3 forms the input to the DETR decoder for bounding box regression and language-token predictions.

2. Mathematical Formulation

Formally, for each input $F$4, MoEE operates as follows:

• Let $F$5 denote the fused vision–text hidden states.
• The attribute-specific mask is $F$6, applied elementwise as $F$7.
• The $F$8-th expert output is $F$9.
• Mean pooling provides summary vectors $H \in \mathbb{R}^{B \times Q \times C}$0.
• The gating network aggregates $H \in \mathbb{R}^{B \times Q \times C}$1, $H \in \mathbb{R}^{B \times Q \times C}$2, projects to logits $H \in \mathbb{R}^{B \times Q \times C}$3, and applies a softmax to yield mixture weights $H \in \mathbb{R}^{B \times Q \times C}$4.
• The fused feature is $H \in \mathbb{R}^{B \times Q \times C}$5.
• Supervision comprises bounding box regression and attribute-aligned token prediction. The per-query loss for query $H \in \mathbb{R}^{B \times Q \times C}$6 and attribute $H \in \mathbb{R}^{B \times Q \times C}$7 is:

$H \in \mathbb{R}^{B \times Q \times C}$8

and the total loss is summed across all matched pairs via Hungarian matching:

$H \in \mathbb{R}^{B \times Q \times C}$9

3. Gating Network Design and Adaptivity

The gating network is central to MoEE's dynamic behavior. Key architectural elements include:

• Input Features: Concatenation of mean-pooled expert summaries $B$0.
• Parameterization: A single linear layer $B$1.
• Stochastic Regularization: Additive Gaussian noise $B$2 (with $B$3 learned end-to-end).
• Soft Selection: Softmax gating to produce $B$4 summing to 1.

Ablation studies demonstrate that gating adapts to event stream properties: low motion (event density) leads to higher $B$5 for appearance and viewer experts, while elevated activity shifts preference toward status and inter-object experts. This behavior is further substantiated by class-specific analysis showing, for example, “rider” and “bike” objects leveraging the status expert, and static “bus” and “truck” categories depending more on appearance and viewer information (Kong et al., 23 Jul 2025).

4. Training Regimen and Objectives

Attribute-aware grounding using MoEE involves dense supervision and robust optimization:

• Supervision: Multi-pseudo-object Hungarian matching allots four pseudo-targets (one per attribute) per true object, enabling dense attribute token alignment without inflating decoder complexity.
• Loss Functions: Weighted combination of L1 box regression, Generalized IoU loss, and cross-entropy over attribute token matches.
• Hyperparameters: Loss weights $B$6 and $B$7 set by validation; no additional auxiliary losses beyond core attribute guidance.
• Optimization: AdamW optimizer (weight decay 0.01), batch size 16, 500-step warmup, cosine decay schedule to 90K steps.

5. Implementation Specifics

The MoEE component is instantiated as follows:

Component	Architecture/Hyperparameters	Learning Rate
Text Encoder	RoBERTa-base, 64 tokens, 768$B$8256 proj.	$B$9
Event Backbone	RVT/FlexEvent Transformer, 9 temporal bins, 256D tokens	$Q=100$0
Frame Backbone	ResNet-101 pretrained, multiscale, 256D	$Q=100$1
Transformer	Encoder: 6L/8H; Decoder: 6L, 100 queries	Init from BUTD-DETR
Gating Network	Linear (4C$Q=100$24), learned $Q=100$3	—
Optimizer	AdamW, weight decay 0.01, batch 16	—

Weights and schedules reflect validation optimization, with all layer weights (except pre-trained RoBERTa and ResNet-101) initialized from a DETR-based grounding model.

6. Empirical Results and Ablation

Extensive ablation studies establish the quantitative impact of MoEE:

• Component Stacking (event-only mAcc @IoU0.95): Baseline (no PWM/MAF/MoEE) yields 22.07%; adding Positive Word Matching (PWM) raises to 26.38%; adding Multi-Attribute Fusion (MAF) brings 27.01%; PWM+MAF achieves 29.66%; and full PWM+MAF+MoEE reaches 31.96% (+9.89).
• Expert Decomposition: Single-attribute experts reach at most 28.90% (status expert); MoEE (all four) reaches 31.96% (+3.98 over best single expert).
• Fusion Approaches: Simple sum yields 26.38%; addition 28.39%; concatenation 27.50%; attention 29.66%; MoEE (softmax+noise) 31.96% (+5.58).
• Task Gains: Frame-only: 55.47% mAcc (EventRefer MoEE) vs. 48.91% (BUTD-DETR), fusion: 61.82% vs. 59.40% (FlexEvent).
• Activation Trends: Class-wise, dynamic classes rely on status expert; static on appearance/viewer; gating dynamically modulates with event density.

7. Context, Relevance, and Significance

MoEE operationalizes interpretable, attribute-attentive fusion for linguistically and temporally grounded event-based perception, addressing the challenge of integrating asynchronous multimodal streams with complex spatial, motion, and relational cues. Experimental evidence supports its consistent superiority over conventional fusions and previous state-of-the-art approaches across multiple data modalities. Notably, MoEE supports explicit dissection of grounding strategies, revealing the contributions of specific attribute cues to object localization and description alignment (Kong et al., 23 Jul 2025).

A plausible implication is that such modular gating strategies could extend to other attribute-centric multimodal grounding tasks, especially where interpretability and adaptivity to sensor dynamics are required. MoEE thus substantiates an approach for precise, context-aware language–scene alignment in complex, real-world dynamic environments.