APRTrack: Robust RGB-Event Tracker

• APRTrack is a robust RGB–Event tracker that addresses modality failure and partial occlusion through adversarial perturbation and hierarchical training.
• It employs a three-way routing mechanism to balance clean and perturbed samples, ensuring feature stability and enhanced localization.
• The method integrates Footprint-guided Channel-calibrated Hopfield Retrieval to restore missing information, achieving competitive benchmark performance.

Searching arXiv for the APRTrack paper and the foundational works explicitly referenced in its description. APRTrack, introduced in "Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking" (Wang et al., 24 Jun 2026), is a single-object tracker for RGB–Event data that explicitly models two common types of real-world degradation—whole-modality failure and local target absence—via adversarial perturbations, and then restores missing information through a controlled associative memory retrieval based on Modern Hopfield Networks. It is designed for RGB-Event tracking, where RGB appearance textures and dense temporal motion cues from event sensors are fused to improve localization robustness, but where harsh environments can make either modality unreliable and can render targets incomplete through occlusion, edge truncation, and foreground clutter.

1. Problem formulation and design objective

APRTrack targets the failure modes that arise when multi-modal fusion is exposed to structured signal degradations rather than idealized noise. The framework is organized around two degradation classes stated explicitly in the paper: full-modal failure and localized target region absence (Wang et al., 24 Jun 2026). This focus distinguishes it from training schemes that rely primarily on random dropout or undifferentiated augmentation.

The central design objective is robustness against partial target missing and modal degradation. A plausible implication is that APRTrack treats missing information not as a purely stochastic nuisance, but as a structured condition that should be synthesized during training and compensated during inference. The paper operationalizes this idea through a hierarchical perturbation and retrieval framework: adversarial perturbation branches generate hard missing-state samples, a hierarchical routing mechanism prevents over-destructive training, and Footprint-guided Channel-calibrated Hopfield Retrieval compensates for missing cues from historical memory (Wang et al., 24 Jun 2026).

APRTrack is therefore best understood as a robustness-oriented RGB-Event tracker whose core contribution is the coupling of active perturbation with controlled memory retrieval. A common misconception would be to interpret it as a generic memory-augmented tracker; in the formulation given by the paper, retrieval is not unconditional, and the historical compensation is explicitly bounded to target regions.

2. Adversarial perturbation branches

APRTrack inserts two adversarial perturbation modules immediately after the shared patch-embedding of the RGB and Event search inputs (Wang et al., 24 Jun 2026). Each branch is driven adversarially via a Gradient Reversal Layer so as to generate the hardest missing-state samples the current backbone can withstand.

For modality-level perturbation, let $X^r, X^e \in \mathbb{R}^{N_x \times D}$ be the RGB and Event search tokens. APRTrack defines a mutually-exclusive gating vector $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$ and applies

$$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$

Instead of random dropout, $\alpha$ is predicted by

$$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$

where $\rho$ aggregates dual-modality context and $\Psi_{\mathrm{mod}}$ is a small MLP that enforces exactly one modality missing or both present. Hard selection is implemented via Gumbel-Softmax. To avoid the trivial solution $\alpha=(1,1)$ or collapse to always dropping the same modality, the model adds the balance regularization

$$\mathcal{L}_{\mathrm{mod\_reg}} = \sum_{m\in\{r,e\}} \left\|\mathbb{E}_i[\alpha_i^m]-\tfrac{1}{2}\right\|^2.$$

For spatial-level perturbation, APRTrack randomly selects one rectangular window $s$ whose area ratio $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$0 lies in a prescribed interval, given in the description as, for example, $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$1. With $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$2 denoting the candidate window mask of occluded tokens, the operator is

$$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$3

The window is itself chosen adversarially: $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$4 where $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$5 scores each patch by its importance for tracking, $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$6 is the soft target mask derived from the ground-truth box, and $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$7 samples a single rectangular region via hard Gumbel-Softmax subject to a constraint on overlap with $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$8. The paper also specifies a regularizer to prevent windows with excessive or insufficient overlap.

These two branches model distinct failure modes. The modality branch simulates complete unreliability of RGB or Event input, whereas the spatial branch simulates local occlusion or truncation. This suggests that APRTrack treats cross-modal absence and intra-target incompleteness as separate sources of brittleness rather than as interchangeable perturbations.

3. Hierarchical routing and training dynamics

Rather than applying both perturbations to every sample, APRTrack uses a three-way routing variable $$\alpha = (\alpha^r,\alpha^e) \in \{(1,1),(1,0),(0,1)\}$$9 that assigns each training sample to exactly one branch (Wang et al., 24 Jun 2026): $$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$0 In the “mod” branch only modality-level perturbation is applied, in “spa” only spatial dropout is applied, and in “clean” none is applied.

The stated purpose of this decoupling is to avoid feature collapse caused by over-destruction. In practical terms, the model is always exposed to at least one unperturbed view during training, which preserves learnable tracking structure while still confronting the backbone with hard missing-state examples. A plausible implication is that hierarchical routing functions as a curriculum over perturbation composition, but the paper formalizes it as branch separation rather than as an auxiliary schedule.

APRTrack also introduces a progressive schedule in the modality branch. At early epochs the adversarial gate is weakened so that only mild modality dropout is applied, and gradually the full adversarial strength is restored. The paper states that this stabilizes joint training of all branches (Wang et al., 24 Jun 2026). This is significant because adversarial sample generation, if introduced too aggressively, can destabilize representation learning when the backbone has not yet formed sufficiently discriminative cross-modal features.

4. Footprint-guided Channel-calibrated Hopfield Retrieval

When the tracker encounters local occlusion in the spatial branch, APRTrack compensates for missing information by retrieving relevant target tokens from a small history. The retrieval module, Footprint-guided Channel-calibrated Hopfield Retrieval, uses the attention-form modern Hopfield update (Ramsauer et al., 2020) as instantiated within APRTrack (Wang et al., 24 Jun 2026).

For each past frame $$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$1, the tracker stores $$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$2, where $$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$3 are the modality-specific tokens and $$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$4 is the soft mask of the target region. These are concatenated over time into historical memories $$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$5, and $$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$6. APRTrack then builds an ROI bias

$$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$7

so that tokens outside the historical target region are heavily penalized in the association logits.

Hopfield association is defined as

$$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$8

To recover RGB cues from history, APRTrack uses

$$\mathcal{P}_{\mathrm{mod}}(X^r,X^e) = (\alpha^r X^r,\alpha^e X^e).$$9

and symmetrically retrieves Event values.

Before full retrieval, APRTrack computes a lightweight association footprint: $\alpha$0 From $\alpha$1, it extracts

$\alpha$2

where

$\alpha$3

is the normalized entropy and

$\alpha$4

is the strongest match. A small MLP $\alpha$5 produces channel-wise weights

$\alpha$6

and the final retrieval becomes

$\alpha$7

Instead of replacing the current tokens $\alpha$8 with $\alpha$9, APRTrack uses gated residual fusion: $$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$0 where $$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$1 is a small network that looks at $$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$2 and decides token-wise how much to trust the retrieved signal. This mechanism is important for interpreting the method correctly: the historical memory complements rather than overwrites the current frame. The paper explicitly argues that footprint-guided calibration and ROI bias make retrieval both reliable and focused on target regions, preventing background drift (Wang et al., 24 Jun 2026).

5. Objective function, architecture, and implementation

The final tracking head follows OSTrack’s one-stream paradigm (Ye et al., 2022), as specified by the APRTrack paper (Wang et al., 24 Jun 2026), and predicts classification scores and 4-D box offsets. The per-sample tracking loss is

$$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$3

The reported settings are $$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$4, $$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$5, and $$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$6.

The full objective is

$$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$7

with small weights $$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$8. The regularization terms are the modality balance term and the spatial overlap penalty preventing trivial windows.

APRTrack uses a Transformer backbone: HiViT-B initialized from MAE pre-training (He et al., 2021). The template and search sizes are $$\alpha = \Psi_{\mathrm{mod}}\bigl(\mathrm{GRL}(\rho(X^r,X^e))\bigr),$$9 and $\rho$0, respectively. The description also specifies patch size $\rho$1, token dimension $\rho$2, and the token counts $\rho$3. Optimization uses AdamW with learning rate $\rho$4, decayed by $\rho$5 at epoch 40, weight decay $\rho$6, batch size 16, and total 50 epochs (Wang et al., 24 Jun 2026).

During training, modality-level perturbation, spatial-level perturbation, and hierarchical routing are enabled, and FCHR learns to retrieve. During inference, modality-level perturbation and spatial-level perturbation are disabled, the historical memory is updated on the fly, and FCHR remains active. This division between training and inference indicates that the perturbation branches are used to shape the robustness properties of the representation rather than to alter the deployed tracking pipeline.

6. Empirical evaluation, interpretation, and scope

APRTrack was trained and evaluated separately on FE108, COESOT, VisEvent, and FELT, using standard Success Rate, Precision Rate, and Normalized Precision Rate, with all results reported on the official test splits (Wang et al., 24 Jun 2026).

Benchmark	APRTrack	Listed comparator(s)
FE108 (108 videos, long-term)	PR $\rho$7, SR $\rho$8	AMTTrack: PR $\rho$9, SR $\Psi_{\mathrm{mod}}$0
COESOT (1 354 videos, 90 categories)	PR $\Psi_{\mathrm{mod}}$1, SR $\Psi_{\mathrm{mod}}$2	AMTTrack: PR $\Psi_{\mathrm{mod}}$3, SR $\Psi_{\mathrm{mod}}$4; ViPT: PR $\Psi_{\mathrm{mod}}$5, SR $\Psi_{\mathrm{mod}}$6
VisEvent (820 videos, short-term)	PR $\Psi_{\mathrm{mod}}$7, SR $\Psi_{\mathrm{mod}}$8	AMTTrack: PR $\Psi_{\mathrm{mod}}$9, SR $\alpha=(1,1)$0
FELT (1 044 videos, long-term with 14 attributes)	PR $\alpha=(1,1)$1, SR $\alpha=(1,1)$2, NPR $\alpha=(1,1)$3	AMTTrack: PR $\alpha=(1,1)$4, SR $\alpha=(1,1)$5, NPR $\alpha=(1,1)$6

Across all datasets, the paper states that APRTrack usually achieves the best PR and very competitive SR, demonstrating especially accurate center localization under severe modality failure or local occlusion. On FELT’s attribute breakdown, APRTrack shows consistent gains under viewpoint change, partial occlusion, deformation, and scale variation, while remaining robust in low illumination and fast-motion scenarios. Qualitatively, it maintains tight, stable bounding boxes when one modality is entirely uninformative, when the target is partially occluded or truncated, and when distractor objects of similar appearance appear (Wang et al., 24 Jun 2026).

The paper’s own interpretation is that the adversarial perturbation branches force the backbone to learn features that do not collapse when one modality vanishes or when only part of the target is visible; that hierarchical routing avoids catastrophic over-destruction in training; and that FCHR’s footprint-guided channel calibration and ROI bias make retrieval reliable and focused on target regions. These claims provide the intended causal account of the reported robustness, although the article’s quantitative evidence is organized at the benchmark level rather than as a formal causal decomposition.

The limitations are explicit. APRTrack does not model very long-term appearance evolution, having no state-space or RNN beyond the memory bank, so in extremely long sequences with gradual drift it may underperform specialized long-term methods. If both modalities and the historical memory become unreliable, the associative retrieval has little to work with. The future extensions proposed in the paper are uncertainty-aware memory updating, stronger temporal dynamics such as state space or Transformer recurrence, and adaptive memory pruning to remove stale or outlier frames (Wang et al., 24 Jun 2026). These limits clarify the scope of the method: APRTrack is optimized for robustness to modality failure and partial occlusion, not for unrestricted long-horizon temporal modeling.