FCHR: Calibrated Hopfield Retrieval for Tracking

• The paper introduces FCHR, which uses a footprint-guided, cross-modal Hopfield mechanism to reliably retrieve historical target cues under local corruption.
• FCHR employs a target-constrained memory with ROI bias to inject useful historical information while suppressing background interference.
• Experimental ablations on datasets like COESOT demonstrate that integrating FCHR into APRTrack improves tracking performance by measurable SR, PR, and NPR metrics.

Footprint-guided Channel-calibrated Hopfield Retrieval (FCHR) is the historical compensation module in APRTrack, a hierarchical perturbation and retrieval framework for RGB-Event visual object tracking that is tailored to robustness against partial target missing and modal degradation (Wang et al., 24 Jun 2026). Within APRTrack, FCHR retrieves useful past target information in a manner that is constrained, reliability-aware, and target-region biased. Its function is not to fill in missing features indiscriminately, but to inject historical cues that are likely to help when the current RGB or Event observation is partially corrupted by occlusion, truncation, clutter, or weak modality response.

1. Problem setting and role in APRTrack

APRTrack is organized around two kinds of structured degradation in RGB-Event tracking: modality-level missing or degradation, and spatial-level missing. The first concerns cases in which one modality becomes largely unusable, such as RGB under low light or overexposure or Event under weak motion or noise. The second concerns cases in which only part of the target is missing, such as occlusion, truncation, or foreground clutter.

FCHR is introduced specifically for the spatial-level case. When the target is only locally corrupted, historical target evidence from previous frames can remain informative, but naïve history retrieval can generate false matches because the current query may be contaminated by background, occluders, or outdated appearance. The module is therefore designed to address partial target missing, occlusion and truncation, background clutter and distractor interference, retrieval unreliability caused by degraded queries, and over-injection of historical features that could overwrite current evidence.

This positioning is central to APRTrack’s division of labor. The perturbation branches model degradation during training, whereas FCHR provides the runtime mechanism that compensates for local corruption by exploiting stored target evidence under explicit constraints.

2. Historical memory, target prior, and cross-modal retrieval

FCHR stores historical information in a target-constrained memory rather than a raw feature pool. For each past time step $\tau < t$, the memory is written as

$$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$

where $K^r_\tau$ denotes historical RGB tokens, $K^e_\tau$ historical Event tokens, and $G_\tau$ a soft target mask from the bounding box. This memory organization already encodes a target prior.

To suppress background retrieval, FCHR defines a region-of-interest bias

$B_{\mathrm{roi}} = \gamma (G - \mathbf{1}),$

so that target-region locations are near zero bias and background locations receive negative bias. This bias is added to the Hopfield logits, thereby focusing memory access on target-related historical regions.

The retrieval direction is explicitly cross-modal:

$$\hat{X}^{r}=\mathcal{H}(Q=X^{e},K=K^{e},V=K^{r};B_{\mathrm{roi}}),$$

$$\hat{X}^{e}=\mathcal{H}(Q=X^{r},K=K^{r},V=K^{e};B_{\mathrm{roi}}).$$

Current Event queries therefore retrieve historical RGB content, and current RGB queries retrieve historical Event content. This configuration uses modality complementarity: whichever current modality is more reliable can query memory and produce compensation for the other branch (Wang et al., 24 Jun 2026).

A common misconception is to treat FCHR as generic memory lookup over past feature maps. The formulation above indicates a narrower mechanism: the memory is target-constrained, retrieval is ROI-biased, and the retrieval direction is deliberately cross-modal.

3. Association footprints and retrieval confidence

Before main retrieval, FCHR computes a lightweight query-memory association footprint:

$$A_f = \mathrm{Softmax}\left(\frac{(QW_q^f)(KW_k^f)^\top}{\sqrt{d}}\right).$$

This footprint is not the final retrieval result. It is an intermediate association map that describes how the query distributes its attention over the memory bank, capturing the shape of the query’s memory match pattern.

From $A_f$, FCHR extracts a reliability descriptor

$$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$0

where $$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$1 is the normalized entropy of the association distribution and $$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$2 is the maximum association probability. These two statistics play complementary roles. Entropy measures dispersion or uncertainty; high entropy indicates that the query matches many memories weakly and thus signals uncertain retrieval. Maximum probability measures peak confidence; a strong maximum suggests a clear target-related memory match.

The ablation reported for APRTrack shows that using entropy plus max probability together is better than either alone. This indicates that FCHR’s “footprint-guided” mechanism is not merely an auxiliary attention map, but a reliability diagnostic that explicitly characterizes both uncertainty and strongest-match confidence (Wang et al., 24 Jun 2026).

A plausible implication is that the footprint functions as a low-cost probe of match quality before expensive or potentially harmful historical compensation is applied.

4. Channel-calibrated metric conditioning

The footprint descriptor is transformed into channel calibration weights:

$$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$3

Retrieval confidence therefore does not directly select a memory item. Instead, it modulates the query channels before the main Hopfield association. High-confidence retrieval yields stronger, better-aligned channel emphasis, whereas low-confidence retrieval yields more conservative channel weighting.

This step is the metric-space adjustment in FCHR. The query is not used as-is; it is reweighted channel-wise according to the reliability inferred from the footprint. The stated effect is that the calibrated query becomes more sensitive to channels aligned with reliable historical target patterns and less sensitive to channels likely contaminated by occlusion or clutter.

In this sense, channel calibration is a pre-Hopfield metric conditioning step rather than retrieval proper. Its role is to reshape the query so that the later memory association becomes more trustworthy. This is also the mechanism by which FCHR achieves controllable historical feature compensation: history is admitted through a reliability-conditioned metric rather than through unconstrained token replacement.

5. Hopfield retrieval and bounded compensation

APRTrack builds on Modern Hopfield Networks and presents both the energy-based form and the attention-like association form. For stored patterns $$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$4 and state $$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$5, the energy is

$$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$6

where $$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$7 is inverse temperature and $$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$8 is a constant independent of $$\mathcal{M}_t = \{(K^r_\tau, K^e_\tau, G_\tau)\}_{\tau<t},$$9. The corresponding iterative update is

$K^r_\tau$0

For query tokens $K^r_\tau$1, keys $K^r_\tau$2, and values $K^r_\tau$3, the standard association is

$K^r_\tau$4

and with bias,

$K^r_\tau$5

FCHR applies this mechanism as

$K^r_\tau$6

The retrieval sequence is therefore: compute the footprint $K^r_\tau$7; summarize reliability with $K^r_\tau$8; compute calibration $K^r_\tau$9; calibrate the query to $K^e_\tau$0; and perform biased Hopfield retrieval using $K^e_\tau$1, memory keys $K^e_\tau$2, and memory values $K^e_\tau$3.

Historical information is then fused through gated residual correction:

$K^e_\tau$4

This equation is consequential because the retrieved content does not overwrite the current representation. Instead, it is injected as a gated residual update. The module thus bounds compensation in three coupled ways: spatially through ROI bias, reliability-wise through footprint-guided calibration, and amplitude-wise through gated residual fusion (Wang et al., 24 Jun 2026).

This directly counters the notion that FCHR performs unrestricted memory substitution. Its design is explicitly bounded to target regions and calibrated by retrieval confidence.

6. Training-time integration and empirical findings

APRTrack uses three branches during training: clean, modality perturbation, and spatial perturbation. The hierarchical routing strategy prevents stacking modality-level and spatial-level degradation on the same sample, because that would overly destroy information. Modality perturbation trains robustness to whole-modality failure, spatial perturbation trains robustness to local target absence, and the clean branch preserves a normal anchor representation.

FCHR is primarily associated with the spatial branch, because historical retrieval is most useful when the target is partially missing but the overall tracking situation remains recoverable. At inference, the perturbation modules are disabled, whereas historical memory update and FCHR remain active. FCHR therefore serves as the runtime compensation mechanism, while the perturbation and routing design serve as the training-time robustness construction mechanism.

The ablation on COESOT isolates four components: modality-level perturbation, spatial-level perturbation, Hopfield retrieval, and footprint-channel calibration. Adding vanilla Hopfield retrieval on top of spatial perturbation improves performance modestly, indicating that historical association helps under local corruption. Adding footprint-channel calibration on top of Hopfield retrieval brings a further gain of +1.0 SR, +1.1 PR, and +1.0 NPR. The paper also reports that using entropy plus max probability as the footprint descriptor is best, and that the full model with all components achieves the best overall performance on COESOT (Wang et al., 24 Jun 2026).

Extensive experiments on FE108, COESOT, VisEvent, and FELT are reported to demonstrate the effectiveness of APRTrack’s strategies for RGB-Event visual object tracking. Within that broader result, the ablation evidence suggests that FCHR’s principal contribution is not memory access alone, but reliability-aware retrieval with calibrated query-space matching.