Memory-assisted Kalman Filter (MeKF)

• Memory-assisted Kalman Filter (MeKF) is a motion-estimation module that enhances traditional Kalman filtering by integrating learned, memory-driven residual corrections.
• It employs LSTM-based memory to capture long-term trajectory history and provides additive compensation in both state prediction and measurement updates.
• Empirical results demonstrate that MeKF significantly reduces drift and improves identity preservation in complex, human-dominant tracking scenarios.

Searching arXiv for papers on "Memory-assisted Kalman Filter (MeKF)" and acronym ambiguity with "MEKF". The Memory-assisted Kalman Filter (MeKF) is a motion-estimation module introduced within MeMoSORT for online multi-person tracking in human-dominant scenarios. It preserves the Bayesian prediction/update structure of Kalman filtering, but augments both the transition model and the measurement model with learned, memory-driven compensation terms derived from trajectory history. In the MeMoSORT formulation, MeKF is intended to address a specific limitation of the standard Kalman filter: the usual first-order Markov, linear motion prior is often too simplistic for human motion in crowded sports and dance scenes, where trajectories can be non-linear, temporally correlated, repetitive, or abruptly changing (Wang et al., 13 Aug 2025).

1. Terminological scope and acronym ambiguity

Within the MeMoSORT framework, the acronym MeKF denotes Memory-assisted Kalman Filter, not Multiplicative Extended Kalman Filter. This distinction matters because the acronym MEKF is already established in estimation literature for multiplicative attitude filtering. In "Alternate Derivation of Geometric Extended Kalman Filter by MEKF Approach" (Chang, 2017), MEKF refers to the multiplicative extended Kalman filter for spacecraft attitude estimation, and the paper derives a geometric multiplicative EKF through a quaternion error-state formulation. Likewise, "Invariant EKF Design for Scan Matching-aided Localization" (Barczyk et al., 2015) uses MEKF as a conventional multiplicative EKF baseline for pose estimation on $SE(3)$, not as a memory-assisted filter. The latter paper explicitly contains no discussion of memory assistance (Barczyk et al., 2015).

In MeMoSORT, by contrast, memory assistance refers to the use of memory-augmented neural networks to compensate for mismatch between assumed and actual object motion. The target state is a 2D detection box,

$\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$

and the task is online estimation of its evolution under noisy detections and association uncertainty. The memory mechanism is introduced because a vanilla Kalman filter predicts the next box from only the immediately previous state, assuming a constant-velocity-like evolution, whereas the MeMoSORT formulation treats human motion as potentially dependent on longer trajectory history (Wang et al., 13 Aug 2025).

2. Standard Kalman baseline and the failure mode MeKF addresses

The standard tracking-by-detection baseline used by MeMoSORT employs a linear measurement model and a first-order Markov motion model. A detection at frame $t$ is written as

$$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$

with measurement equation

$$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$

and motion equation

$$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$

In the standard Kalman filter recursion, the state is predicted using $\mathbf{F}$, the covariance is predicted using $\mathbf{F}\mathbf{P}\mathbf{F}^T + \mathbf{Q}$, the Kalman gain is computed, and the prediction is corrected with the measurement residual (Wang et al., 13 Aug 2025).

The MeMoSORT paper identifies the main deficiency of this baseline as motion mismatch. Real motion is described as neither strictly first-order Markovian nor fully linear. The paper gives examples such as coordinated or repetitive behavior, including a dancer performing a repeated spinning action after a jump. In such cases, the next state depends on more than just $\bm{b}_{t-1}$, the true dynamics may exhibit long-term temporal dependencies, and even the effective observation process may deviate from the ideal linear measurement map. This is the motivation for replacing the strict one-step Markov assumption with a learned non-Markovian approximation based on stored trajectory history (Wang et al., 13 Aug 2025).

3. Non-Markovian formulation and memory-augmented filtering equations

The MeMoSORT formulation introduces a non-Markovian motion model

$$\bm{b}_t = f_t(\bm{b}_{t-1}, \bm{b}_{t-2}, \ldots, \bm{b}_1) + \bm{w}_t,$$

where $\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$0 is a non-linear transition function depending on the full trajectory history. Because directly modeling $\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$1 is described as intractable, the paper defines a memory state

$\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$2

so that memory compresses the past trajectory into a recurrent state. A compensation term is then generated from memory,

$\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$3

yielding the approximate motion equation

$\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$4

The measurement process is also compensated: $\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$5 MeKF is therefore a Kalman filter with learned additive corrections to both motion prediction and measurement update (Wang et al., 13 Aug 2025).

The paper implements the abstract functions $\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$6, $\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$7, and $\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$8 with neural networks organized into three components.

Memory Update Gate (MUG). The memory update function is implemented using an LSTM: $\bm{b}_t = [x_t, y_t, w_t, h_t]^T,$9 where $t$0 is the LSTM cell state, $t$1 is the hidden state, and $t$2 is the previous memory. The recurrent memory is intended to encode historical motion information rather than appearance features; the stated interpretation is that it distills trajectory history into a compact latent representation encoding motion trends, turning behavior, repetitive movement, and other long-range patterns.

State Prediction Gate (SPG). The paper explicitly retains the physical prior and adds only a residual correction: $t$3

$t$4

The learned correction and covariance compensation are produced by two distinct MLPs,

$t$5

$t$6

The paper stresses that MeKF does not discard the physical model; $t$7 remains the baseline prediction, and the network only predicts a residual correction.

State Update Gate (SUG). The measurement update is also compensated: $t$8

$t$9

$$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$0

The learned terms are

$$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$1

$$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$2

The effective residual is therefore not merely detection minus predicted observation; it is the compensated term

$$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$3

This is the mechanism by which MeKF attempts to explain away systematic observation mismatch before applying Kalman correction (Wang et al., 13 Aug 2025).

4. Online recursion and operational interpretation

Within online tracking, the MeKF recursion is described as operating on the previous track state $$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$4, previous covariance $$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$5, previous memory $$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$6, LSTM states $$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$7, and the current detection $$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$8 if one is associated. The first stage is memory update through the LSTM. The second stage is compensated motion prediction through the SPG, which outputs both a residual motion correction and a covariance compensation term. The third stage, when an associated detection exists, uses the SUG to compute a compensated measurement correction, the Kalman gain, the updated state, and the updated covariance (Wang et al., 13 Aug 2025).

The MeMoSORT description gives the following stepwise interpretation.

1. Compute the updated memory state

$$\widetilde{\bm{b}}_t = [\widetilde{x}_t, \widetilde{y}_t, \widetilde{w}_t, \widetilde{h}_t]^T,$$9

2. Predict motion using memory-derived residuals

$$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$0

followed by

$$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$1

3. If a detection is assigned to the track, compute measurement compensation

$$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$2

and then apply the Kalman-style update.

The outputs are the updated track state $$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$3, the updated covariance $$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$4, and the updated memory $$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$5. If the track is not matched, the filter relies on the prediction step and the memory continues to evolve according to the track’s retained history (Wang et al., 13 Aug 2025).

This operational description clarifies that MeKF remains a recursive filter rather than a direct sequence-to-sequence predictor. The paper explicitly characterizes it as still Bayesian / filter-based, in contrast to trackers that directly predict states with a neural network (Wang et al., 13 Aug 2025).

5. Memory contents, uncertainty modeling, and implementation characteristics

The memory used by MeKF is not defined as a key-value database or external differentiable memory. The paper instead specifies an internal recurrent memory implemented with LSTM states. What is stored is a compact representation of the object’s past motion trajectory, temporal patterns extracted from previous states, and, implicitly, information about velocity trends, turning, periodicity, and trajectory shape. Memory is read through the MLPs that output compensation terms for prediction and covariance, and it is written through the LSTM update at each step using previous hidden and cell states together with the retained trajectory history (Wang et al., 13 Aug 2025).

A second defining characteristic is that MeKF modifies not only the state prediction but also the uncertainty model. Standard Kalman filtering uses only $$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$6 and $$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$7, whereas MeKF adds learned covariance terms $$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$8 and $$\widetilde{\bm{b}}_t = \mathbf{H}\bm{b}_t + \bm{v}_t, \qquad \bm{v}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{R}_t),$$9. The concrete distinction given by the paper is therefore fivefold: non-Markovian motion modeling, a learned memory state, learned residuals in both prediction and update, learned covariance compensation, and preservation of the Kalman prediction/update cycle (Wang et al., 13 Aug 2025).

The available implementation details are limited but explicit. The paper states that MeKF training uses AdamW with learning rate $$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$0. It also reports that the detector is YOLOX fine-tuned on CrowdHuman and the target dataset, and that all other unspecified hyperparameters and ReID settings are kept consistent with Deep OC-SORT. The excerpted material does not provide a detailed explicit loss formula for MeKF in the main text, although it states that the gates are “supported by theoretical derivations” in Appendix A (Wang et al., 13 Aug 2025).

Two misconceptions are directly addressed by the formulation. First, MeKF does not replace the physical model with a purely learned transition; the network supplies only residual corrections to a baseline $$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$1-based prediction. Second, the memory is motion-centric rather than appearance-centric: the paper explicitly says that the intended contents of memory are not appearance features but historical motion information (Wang et al., 13 Aug 2025).

6. Empirical behavior and relation to adjacent Kalman-filter variants

The MeMoSORT paper reports that experiments on DanceTrack and SportsMOT show that the full tracker achieves HOTA scores of $$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$2 and $$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$3, respectively (Wang et al., 13 Aug 2025). These numbers are reported for the overall tracker, whose two stated innovations are MeKF and the Motion-adaptive IoU (Mo-IoU). For MeKF specifically, the ablation study is more informative about its isolated role. The paper reports a baseline tracker without the proposed motion model at 56.94 HOTA, and the version with MeKF at 67.41 HOTA, corresponding to about $$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$4 HOTA in the component ablation. In the motion-model ablation comparing None, KF, LSTM, Transformer, Diffusion, and MeKF, the MeKF variant performs best overall, with

$$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$5

The qualitative case study described in the paper attributes improved identity preservation in complex-motion scenarios to the “precise state estimation” produced by MeKF (Wang et al., 13 Aug 2025).

These empirical results position MeKF as a hybrid between classical Bayesian filtering and learned sequence modeling. A plausible implication is that its primary benefit arises when the rigid constant-velocity extrapolation of standard tracking KFs becomes misaligned with motion patterns that exhibit recurrence or long-range temporal structure. The paper itself formulates this more narrowly: MeKF is intended to reduce drift during difficult nonlinear motion, keep predicted boxes close enough to detections to preserve identity, and improve downstream association quality (Wang et al., 13 Aug 2025).

The relation to other Kalman-filter variants is mostly terminological rather than methodological. In attitude estimation and localization, MEKF usually denotes the Multiplicative Extended Kalman Filter, which uses multiplicative error states on manifolds such as $$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$6 or $$\bm{b}_t = \mathbf{F}\bm{b}_{t-1} + \bm{w}_t, \qquad \bm{w}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t).$$7. The spacecraft-attitude paper (Chang, 2017) derives a geometric multiplicative EKF with quaternion-based attitude correction and a geometric gyroscope-bias update. The scan-matching localization paper (Barczyk et al., 2015) compares an IEKF with a multiplicative EKF baseline and concludes that the IEKF is more robust on nonlinear trajectories. Neither paper addresses memory augmentation, learned residual correction, LSTM-based trajectory memory, or neural covariance compensation. Consequently, although the acronyms are visually similar, MeKF in MeMoSORT belongs to a different line of work: memory-augmented tracking filters rather than multiplicative geometric EKFs (Chang, 2017, Barczyk et al., 2015).