Event Transition Tensors in SSMs

• Event Transition Tensors are dynamic, event-adaptive transition matrices that modulate state changes in SSMs based on real-time event density.
• They leverage a trainable modulation matrix and an exponentially weighted moving average to balance rapid motion and stasis, mitigating underfitting and overfitting.
• Integrated in frameworks like Vision Mamba, they enhance multimodal RGB-event object tracking by preserving baseline SSM priors while enabling adaptive temporal evolution.

Event Transition Tensors are dynamic, event-adaptive transition matrices introduced within State Space Models (SSMs) to enhance temporal modeling of multimodal signals, particularly in RGB-Event object tracking. In contrast to conventional SSMs that employ a static state-transition matrix, Event Transition Tensors—formulated as a time-varying transition tensor—modulate the state transitions on a per-timestep basis according to real-time event density. This mechanism enables differentiated temporal evolution for sparse and dense event streams, directly addressing issues of underfitting and overfitting in cross-modal tracking pipelines, such as those built on Vision Mamba frameworks (You et al., 15 Apr 2026).

1. Formulation of Event Transition Tensors in State Space Models

Let the hidden state at timestep $t$ be $x_t \in \mathbb{R}^D$, and the input $u_t \in \mathbb{R}^D$ (typically after a Conv1d or linear projection). Standard linear SSMs utilize the update:

$$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$

where $A \in \mathbb{R}^{D \times D}$ is a static transition matrix, and $B, C, D$ are learned projections. In Event Transition Tensor systems—specifically in Dynamic SSM (DSSM) layers—the static $A$ is replaced by a tensor $T_t$ (written as $A_t$ in the reference work), resulting in the adaptive update:

$x_t = T_t x_{t-1} + B u_t$

This enables the transition dynamics to be modulated on-the-fly as a function of the local event stream density.

2. Event Density Measurement and Modulation Matrix Construction

The mechanism begins with the definition of scalar event-density $x_t \in \mathbb{R}^D$0 computed over the spatial region ($x_t \in \mathbb{R}^D$1) aligned to the current RGB frame:

$x_t \in \mathbb{R}^D$2

where $x_t \in \mathbb{R}^D$3 enumerates events in window $x_t \in \mathbb{R}^D$4. To map $x_t \in \mathbb{R}^D$5 into an actionable transition modulation, a trainable weight matrix $x_t \in \mathbb{R}^D$6 computes a D-by-D modulation matrix $x_t \in \mathbb{R}^D$7:

$x_t \in \mathbb{R}^D$8

where $x_t \in \mathbb{R}^D$9 denotes element-wise sigmoid activation, ensuring $u_t \in \mathbb{R}^D$0 and constraining the dynamic scaling to prevent numerical instability under fluctuating event counts.

3. State Transition Update and Bias Injection

The constructed modulation $u_t \in \mathbb{R}^D$1 forms the basis of an exponentially weighted moving average for the dynamic transition. Let $u_t \in \mathbb{R}^D$2 (typically the identity $u_t \in \mathbb{R}^D$3) provide a fixed prior, and $u_t \in \mathbb{R}^D$4 be a learnable scalar controlling the temporal mixing. The recurrent update is:

$u_t \in \mathbb{R}^D$5

This exponentially smooth blend ensures that abrupt surges in event density do not result in catastrophic shifts in the transition dynamics but rather allow the model to adapt smoothly, reflecting both instantaneous and preceding temporal information.

To restore compatibility with the base SSM structure found in Vision Mamba layers, the resultant adaptive transition $u_t \in \mathbb{R}^D$6 is not used in place of $u_t \in \mathbb{R}^D$7 but is instead injected as a bias:

$u_t \in \mathbb{R}^D$8

This additive biasing preserves the SSM’s architectural inductive priors while facilitating event-adaptive modulation.

4. Tensor Shapes and Implementation Notes

The dimensionalities involved in the Event Transition Tensor mechanism are summarized as follows:

Symbol	Shape	Description
$u_t \in \mathbb{R}^D$9, $$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$0, $$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$1	$$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$2	State/input/output (post-projection)
$$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$3, $$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$4, $$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$5, $$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$6, $$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$7, $$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$8	$$x_t = A x_{t-1} + B u_t, \qquad y_t = C x_t + D u_t$$9	Transition/projection/modulation matrices

All dynamic transitions $A \in \mathbb{R}^{D \times D}$0 are square matrices matching the hidden-state dimension of the SSM layer. Event density-modulated $A \in \mathbb{R}^{D \times D}$1 and all trainable parameterizations maintain strict dimensionality consistency with the SSM’s internal representations.

5. Regularization, Normalization, and Stability Considerations

Two primary regularization strategies enforce numerical stability and modeling robustness:

• The element-wise sigmoid on $A \in \mathbb{R}^{D \times D}$2 bounds dynamic scaling within $A \in \mathbb{R}^{D \times D}$3, thereby averting potential spikes from abrupt event bursts.
• The use of a moving average in forming $A \in \mathbb{R}^{D \times D}$4 ($A \in \mathbb{R}^{D \times D}$5) ensures continuity between consecutive state-transitions, reducing volatility due to short-term event surges and enforcing temporal coherence in system dynamics.

Injecting $A \in \mathbb{R}^{D \times D}$6 as a bias term, rather than wholly supplanting $A \in \mathbb{R}^{D \times D}$7, aligns dynamic adaptation with the underlying structural priors of Vision Mamba SSMs, which empirically supports preservation of baseline performance and interpretability.

6. Theoretical Motivation and Empirical Benefits

Static state transition matrices are fundamentally limited, as they cannot discriminate between slow, sparse-event (risk of underfitting) and fast, dense-event (risk of overfitting) regimes. By adaptively modulating the transition tensor $A \in \mathbb{R}^{D \times D}$8 with the observed event-density $A \in \mathbb{R}^{D \times D}$9, the system dynamically accelerates temporal evolution during periods of high motion (many events) and decelerates during stasis (few events). This behavior directly addresses tracking drift during rapid object motion and improves resource allocation by mitigating unnecessary modeling when targets are stationary.

Empirical results, as reported for FE108 and FELT datasets, demonstrate that event transition tensors implemented within the MambaTrack framework yield more robust, flexible cross-modal fusion and state modeling when compared to any fixed-$B, C, D$0 baseline (You et al., 15 Apr 2026).

7. Applications and Integration in Multimodal Tracking Systems

Event Transition Tensors have been deployed within object-tracking frameworks that fuse RGB and event camera data streams, such as MambaTrack. Beyond simply governing temporal evolution, these tensors act as a foundation for robust multimodal integration via downstream modules (e.g., Gated Projection Fusion). Their lightweight construction and compatibility with real-time SSM implementations make them suitable candidates for embedded and on-system deployment in resource-constrained visual tracking scenarios (You et al., 15 Apr 2026).