Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking

Abstract: RGB-Event tracking improves localization robustness by fusing RGB appearance textures and dense temporal motion cues from event sensors. While this multi-modal scheme broadens tracking applicability, real-world scenes suffer diverse structured signal degradations that hinder traditional multi-modal fusion. In harsh environments, either modality can lose reliability drastically, and targets frequently appear incomplete due to occlusion, edge truncation and foreground clutter.To tackle the above challenges, we present a hierarchical perturbation and retrieval framework tailored for RGB-Event tracking with robustness against partial target missing and modal degradation, termed APRTrack. To mimic real-world signal corruption, APRTrack constructs structured degradation via two adversarial perturbation branches at the modality and spatial levels, which separately simulate full-modal failure and localized target region absence. A hierarchical routing mechanism is designed to disentangle the training pipelines of the two perturbation types, effectively eliminating feature collapse induced by superimposed degradation constraints. Furthermore, we devise Footprint-guided Channel-calibrated Hopfield Retrieval (FCHR) for reliable historical information compensation. This module evaluates retrieval confidence based on association footprints between queries and memory banks, and calibrates the retrieval metric space prior to Hopfield matching, realizing controllable historical feature compensation bounded to target regions. Extensive experiments on FE108, COESOT, VisEvent, and FELT datasets demonstrate the effectiveness of our proposed strategies for the RGB-Event visual object tracking. The source code and pre-trained models will be released on https://github.com/Event-AHU/OpenEvTracking

• The paper introduces APRTrack, which employs adversarial perturbations and memory-driven retrieval to handle modality failures and spatial occlusions.
• It utilizes a Transformer-based pipeline with hierarchical routing to simulate real-world missing scenarios, enhancing tracking robustness.
• Extensive experiments demonstrate state-of-the-art precision and real-time performance across diverse benchmarks, validating its practical impact.

Active Adversarial Perturbation-Driven Associative Memory Retrieval for RGB-Event Visual Object Tracking

Introduction and Motivation

RGB-Event visual object tracking synergizes the rich appearance textures from conventional RGB sensors with dense temporal motion cues from event-based sensors to achieve robust target localization under diverse and challenging conditions. However, real-world deployments frequently encounter structured signal degradations—ranging from modality-level sensor failure to localized occlusions and edge truncations—leading to severe input corruption (Figure 1). These pose unique challenges to traditional multi-modal fusion architectures, which often presume intact, reliable inputs and fail to generalize in environments where one or both modalities become unreliable or spatially incomplete.

Figure 1: Representative modality-level and spatial-level missing issues in RGB-Event tracking.

APRTrack is introduced to address these structured degradations, employing adversarial hierarchical perturbation and memory-driven retrieval mechanisms to enhance tracking robustness. The framework actively simulates complex real-world missing patterns during training and leverages association-driven memory compensation to mitigate performance drops from missing modalities and local target absence.

APRTrack Architecture and Methodology

APRTrack is anchored on a Transformer-based tracking pipeline and innovatively injects missing-robustness mechanisms immediately after token embedding and prior to relation modeling and localization (Figure 2). The approach comprises three principal modules:

Figure 2: APRTrack overview, including hierarchical perturbation and footprint-guided Hopfield retrieval for robust RGB-Event tracking.

Hierarchical Adversarial Perturbation

Dual perturbation branches are established:

• Modality-level Adversarial Perturbation: Simulates complete modality failure (either RGB or Event) by learnable, mutually exclusive gating. Unlike prior random dropout strategies, APRTrack adaptively selects challenging missing states during training via adversarial optimization, ensuring the model learns to localize with one available modality.
• Spatial-level Adversarial Continuous Perturbation: Simulates local target absence through adversarial scoring and window sampling, generating occlusion masks tightly aligned with target regions. This approach models realistic occlusion geometry, avoiding semantically meaningless token dropout.

A hierarchical routing mechanism assigns samples to clean, modality, or spatial degradation branches, ensuring that degradation types are handled independently, preventing compounded feature collapse.

Footprint-Guided Channel-Calibrated Hopfield Retrieval (FCHR)

To counter local target absence, APRTrack integrates a cross-modal associative memory retrieval module, inspired by Modern Hopfield Networks. FCHR organizes historical frame-token memories with ROI bias, enabling retrieval primarily from target regions (Figure 3).

Figure 3: Architecture of query-memory association footprint estimation, channel calibration, and ROI-biased Hopfield retrieval.

Association footprints—composed of entropy and maximal match statistics—quantify retrieval reliability and drive dynamic channel calibration, shaping the query space before Hopfield association. Compensation is injected conditionally via gated residual fusion, safeguarding against erroneous memory overwrites. This ensures historical features only augment current corrupted representations when high confidence in retrieval exists.

Experimental Analysis

APRTrack is evaluated across FE108, COESOT, VisEvent, and FELT benchmarks, with metrics including Success Rate (SR), Precision Rate (PR), and Normalized Precision Rate (NPR). Results consistently indicate robust improvements in localization accuracy and success under challenging degradation regimes. Notably, APRTrack achieves PR scores of 97.0 on FE108, 84.0 on COESOT, 79.4 on VisEvent, and 70.1 on FELT with competitive SR/NPR, outperforming state-of-the-art baselines in both short-term and long-term tracking scenarios.

Attribute-level analysis on FELT (Figure 4) reveals sustained gains across viewpoint transformation, occlusion, deformation, and scale variation, emphasizing the framework's capacity to generalize robustness across diverse degradation modes.

Figure 4: Success rate comparison under 14 challenging attributes on FELT.

Ablation studies validate the contribution of each module: modality-level perturbation, spatial-level perturbation, Hopfield retrieval, and footprint-channel calibration are quantitatively shown to be complementary, yielding cumulative improvements. Hierarchical routing is critical for stable optimization, and progressive modality perturbation scheduling further ameliorates early-stage destabilization.

Efficiency evaluations demonstrate that computational overhead introduced by FCHR remains moderate (parameter count increases from 70.65M to 81.86M, inference FLOPs from 56.87G to 78.50G), with real-time capability maintained at 31 FPS.

Dynamics and Visualization

Compensation gate dynamics (Figure 5) show adaptive compensation injection, particularly higher for Event streams, reflecting their susceptibility to target sparsity under weak motion. Visualization of attention and response maps (Figures 6 and 7) illustrate APRTrack’s capacity to focus on discriminative target regions even under distractor conditions and localized corruption; response maps confirm stable target prediction under fast motion and occlusion.

Figure 5: Compensation gate dynamics of FCHR during training.

Figure 6: Visualization of attention maps generated by APRTrack.

Figure 7: Visualization of response maps generated by APRTrack.

Qualitative comparisons (Figure 8) demonstrate the superiority of APRTrack across sequences with severe modality and spatial missing, yielding consistent bounding box stability.

Figure 8: Tracking results visualization with APRTrack and other SOTA trackers.

Practical Implications and Future Directions

APRTrack’s methodology fundamentally shifts RGB-Event tracking from naive fusion toward an adversarial, compensation-centric paradigm that actively prepares models for structured missing scenarios. By explicitly disentangling modality and spatial degradation and integrating associative memory mechanisms, this framework lays the groundwork for missing-robust multi-modal systems applicable in surveillance, autonomous robotics, and intelligent unmanned platforms.

The footprint-guided memory retrieval approach opens avenues for further reliability-aware compensation strategies, especially in longer-term sequences exhibiting significant appearance drift. Integrating uncertainty modeling and dynamic memory updating mechanisms could strengthen robustness in exceedingly degraded regimes where both modalities or memory are corrupted.

Conclusion

APRTrack exemplifies a principled approach to missing-robust RGB-Event visual object tracking, leveraging hierarchical adversarial perturbation and footprint-guided memory retrieval for controlled, confidence-aware historical compensation. Extensive empirical results demonstrate consistent improvements over the state-of-the-art across diverse benchmarks and degradation modes. The framework’s modularity and theoretical underpinnings offer a promising foundation for future exploration in robust multi-modal visual tracking and associative memory retrieval models in AI systems.