MeMoSORT: Memory-Assisted Multi-Object Tracking

• The paper introduces MeMoSORT, a method that enhances the conventional Kalman filter with memory-assisted residual corrections to better predict nonlinear human motion.
• It employs Motion-adaptive IoU by expanding detection boxes and modulating height similarity to improve association under occlusion and abrupt movements.
• Empirical results on DanceTrack and SportsMOT show significant gains over baselines, maintaining real-time performance despite the added complexities.

MeMoSORT is an online, real-time tracking-by-detection method for multi-person and multi-object tracking in human-dominant scenes, introduced as “Memory-Assisted Filtering and Motion-Adaptive Association Metric for Multi-Person Tracking” (Wang et al., 13 Aug 2025). It targets settings such as DanceTrack and SportsMOT, where appearance is often weakly discriminative, motion is nonlinear and abrupt, and severe occlusion is frequent. Its two defining components are the Memory-assisted Kalman filter (MeKF), which augments a conventional Kalman filter with learned residual motion and measurement corrections, and Motion-adaptive IoU (Mo-IoU), which relaxes rigid box-overlap matching by expanding the admissible spatial region and modulating the role of height similarity according to motion.

1. Problem setting and rationale

MeMoSORT is designed for the regime in which the conventional SORT/ByteTrack-style recipe—linear Kalman prediction followed by IoU-based association—becomes the dominant source of failure rather than the detector itself (Wang et al., 13 Aug 2025). The motivating claim is that, in human-dominant scenes, a standard Kalman filter is often mismatched to actual object dynamics because it assumes a linear first-order Markov model, whereas dancers and athletes exhibit coordinated, repetitive, variable-speed, and nonlinear trajectories. The resulting prediction bias accumulates into filtering error, which then degrades association. In parallel, standard IoU matching is described as spatially rigid: after abrupt motion or occlusion, the predicted box and the correct detection may have very small or zero overlap, producing association failure, identity switches, or target loss.

This design position places MeMoSORT within the broader SORT lineage but with a different diagnosis of the main bottleneck. Deep SORT, for example, preserved SORT’s online Kalman-filter-plus-Hungarian structure while adding an offline-trained deep association metric and reduced identity switches by 45% relative to SORT on MOT16 (Wojke et al., 2017). MeMoSORT instead argues that in DanceTrack- and SportsMOT-like settings, motion-model mismatch and rigid spatial association remain central weaknesses even when appearance cues are available (Wang et al., 13 Aug 2025).

A common misconception is to interpret MeMoSORT as a fully neural or end-to-end tracker. It is not. The method explicitly keeps the tracking-by-detection pipeline—detect, predict, associate, update, manage tracks—but modifies the prediction/update model and the main association metric. This architectural conservatism is central to its stated goals of simplicity, online operation, and real-time speed (Wang et al., 13 Aug 2025).

2. State-space formulation and the Memory-assisted Kalman filter

The baseline formulation is a conventional linear state-space model over bounding boxes. A detection at frame $t$ is represented as

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$

with observation model

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$

and motion model

$$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$

MeMoSORT starts from this model but asserts that true motion is better viewed as non-Markovian:

$$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$

Because the explicit form of $f_t(\cdot)$ is treated as intractable, the method introduces a memory state $\mathbf{m}_t$ that summarizes motion history:

$$\mathbf{m}_t = \phi_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1)\approx \phi_t(\mathbf{b}_{t-1},\mathbf{m}_{t-1}).$$

This memory generates a compensation term

$\Delta_t^{\mathbb{F}} = \psi_t(\mathbf{m}_t),$

so that prediction becomes

$$\mathbf{b}_t \approx \mathbf{F}\mathbf{b}_{t-1} + \Delta_t^{\mathbb{F}} + \mathbf{w}_t.$$

The measurement model is likewise compensated:

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$0

MeKF implements this idea with three gates. The Memory Update Gate uses an LSTM to maintain the historical representation,

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$1

where $$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$2 and $$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$3 are the LSTM cell and hidden states. The State Prediction Gate modifies both the predicted mean and covariance:

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$4

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$5

with

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$6

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$7

The State Update Gate analogously adapts the observation correction and uncertainty:

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$8

$$\widetilde{\mathbf{b}}_t = [\widetilde{x}_t,\widetilde{y}_t,\widetilde{w}_t,\widetilde{h}_t]^T,$$9

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$0

where

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$1

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$2

The method therefore learns four residual corrections: prediction mean, prediction covariance, measurement mean, and measurement covariance. This is a distinguishing feature of MeKF. It does not replace Bayesian filtering with a purely learned trajectory predictor; rather, it retains the Kalman prior and uses the neural modules to compensate for model mismatch. This suggests a hybrid design philosophy: the linear filter remains the fallback structure, while the learned memory handles systematic deviation from that prior (Wang et al., 13 Aug 2025).

3. Motion-adaptive IoU and association

The second major component is Motion-adaptive IoU, introduced to replace rigid IoU in the primary high-confidence association stage (Wang et al., 13 Aug 2025). It is defined as

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$3

The first factor, Expansion IoU, computes IoU after symmetric box enlargement:

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$4

with expanded boxes

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$5

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$6

When $$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$7, EIoU reduces exactly to standard IoU. For larger $$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$8, the matching region expands, making association more tolerant to prediction drift and localization error.

The second factor, Height IoU, emphasizes vertical extent similarity:

$$\widetilde{\mathbf{b}}_t = \mathbf{H}\mathbf{b}_t + \mathbf{v}_t,\qquad \mathbf{v}_t \sim \mathcal{N}(\mathbf{0},\mathbf{R}_t),$$9

where $$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$0 is the intersection height of $$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$1 and $$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$2. The motivation is explicitly human-centric: under crowding and partial occlusion, full 2D overlap may be unreliable, but height and vertical alignment can remain informative.

The “motion-adaptive” element is the selection of $$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$3 and $$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$4 from normalized motion descriptors:

$$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$5

$$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$6

where

$$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$7

The intended interpretation is direct. Faster center motion implies larger likely prediction error and therefore a more tolerant expansion parameter, while rapidly changing height implies that height similarity is less reliable and should be weighted differently. The paper states that these parameters are quantized to discrete levels for efficiency rather than continuously modulated (Wang et al., 13 Aug 2025).

4. Online pipeline, training, and implementation

The online inference pipeline is causal and follows a multi-stage association structure (Wang et al., 13 Aug 2025). YOLOX generates detections. Existing tracks are propagated by MeKF, yielding $$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$8 and $$\mathbf{b}_t = \mathbf{F}\mathbf{b}_{t-1} + \mathbf{w}_t,\qquad \mathbf{w}_t \sim \mathcal{N}(\mathbf{0},\mathbf{Q}_t).$$9. Detections are then split by confidence into high-score and low-score groups using thresholds $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$0 and $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$1. High-score detections are matched in a Deep OC-SORT-like stage using a cost that combines Mo-IoU and ReID; low-score detections are used only in a subsequent standard-IoU stage for the remaining unmatched pairs. Hungarian assignment is applied. Matched tracks are updated through the State Update Gate in MeKF, and unmatched detections may initialize new tracks.

This means that MeMoSORT’s novelties are concentrated in two places rather than in track lifecycle design. Prediction and update are modified through MeKF, and the main association cost is modified through Mo-IoU. The paper states that unspecified settings remain consistent with Deep OC-SORT, so confirmation, lost/removed states, and related management rules are inherited rather than redesigned (Wang et al., 13 Aug 2025).

Training is only partially specified. The detector is not jointly optimized end to end with the tracker. YOLOX is fine-tuned from COCO pretraining on CrowdHuman and the target dataset, “similar to SportsMOT.” MeKF is trained with AdamW and learning rate $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$2. The available text does not provide explicit loss equations for MeKF, batch size, epoch count, sequence length, or network dimensions. For Mo-IoU, the adaptation thresholds are chosen from the training-set velocity distributions: the 50th and 70th percentiles of normalized motion. These are $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$3 and $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$4 on DanceTrack, and $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$5 and $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$6 on SportsMOT. The text reports $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$7, $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$8, $$\mathbf{b}_t = f_t(\mathbf{b}_{t-1},\mathbf{b}_{t-2},\ldots,\mathbf{b}_1) + \mathbf{w}_t.$$9, and $f_t(\cdot)$0, while also noting that the surrounding prose is slightly garbled (Wang et al., 13 Aug 2025).

Real-time operation is treated as a practical constraint rather than a rhetorical claim. On DanceTrack validation, the baseline runs at 74.5 FPS, adding MeKF yields 60.8 FPS, adding Mo-IoU yields 49.4 FPS, and adding ReID yields 28.8 FPS. Each added module reduces throughput, but the system remains real-time according to the paper’s reported measurements (Wang et al., 13 Aug 2025).

5. Empirical results and ablation evidence

The principal benchmarks are DanceTrack and SportsMOT, both chosen because they stress motion handling and long-term association more than appearance discrimination (Wang et al., 13 Aug 2025).

Benchmark	HOTA	AssA	IDF1	DetA	MOTA
DanceTrack test	67.9	54.3	68.0	85.0	93.4
SportsMOT test	82.1	75.6	86.4	89.3	97.0

On DanceTrack test, MeMoSORT exceeds the prior best HOTA reported in the comparison table, TrackTrack at 66.5, and also surpasses Deep OC-SORT at 61.3, DiffMOT at 62.3, and UCMCTrack at 63.6. On SportsMOT test, it improves on Deep HM-SORT at 80.1 HOTA and SportMamba at 77.3 HOTA; the paper highlights gains over SportMamba of $f_t(\cdot)$1 HOTA, $f_t(\cdot)$2 AssA, and $f_t(\cdot)$3 IDF1. These gains are interpreted in the paper as evidence that the hybrid design—Kalman prior plus learned residual correction plus motion-adaptive association—can outperform heavier fully data-driven motion models in these settings (Wang et al., 13 Aug 2025).

The ablations are especially central to the method’s argument. On DanceTrack validation, the baseline reaches 56.94 HOTA. Adding MeKF increases HOTA to 67.41, a gain of $f_t(\cdot)$4. Adding Mo-IoU on top of MeKF increases HOTA further to 77.54, a gain of $f_t(\cdot)$5. Adding ReID yields only a marginal increase to 77.91. This result is used to support the claim that motion modeling and association quality matter more than appearance on this benchmark (Wang et al., 13 Aug 2025).

Filter ablations under the same framework also favor MeKF. Reported HOTA values are 55.24 with no motion model, 56.94 with a standard KF, 60.16 with LSTM, 64.12 with Transformer, 65.91 with Diffusion, and 67.41 with MeKF. MeKF also achieves the best AssA at 49.58, IDF1 at 66.41, and MOTA at 97.55, while its DetA at 91.69 is slightly below some alternatives. Association-metric ablations show a consistent progression: replacing standard IoU by EIoU gives 70.80 HOTA, HMIoU gives 72.70, HA-EIoU gives 75.21, and full Mo-IoU reaches 77.54. This sequence is the empirical basis for the claim that both expansion and height cues are useful, and that motion-adaptive parameter selection is superior to static variants (Wang et al., 13 Aug 2025).

6. Position within memory-based MOT research

MeMoSORT belongs to the memory-based MOT literature, but its notion of memory is specific. It is not a per-track appearance gallery in the Deep SORT sense, where each identity stores up to the last 100 appearance descriptors and association uses nearest-neighbor cosine distance (Wojke et al., 2017). Nor is it an end-to-end Transformer tracker with an explicit spatio-temporal embedding memory bank, as in MeMOT, which stores track histories in a tensor $f_t(\cdot)$6 and performs joint detection and association through memory-encoded queries (Cai et al., 2022).

Instead, MeMoSORT preserves the classical tracking-by-detection skeleton and introduces memory primarily as a recurrent mechanism for compensating motion-model mismatch (Wang et al., 13 Aug 2025). This suggests that “memory” in MeMoSORT is operationally closer to learned residual state correction than to explicit long-range identity retrieval. Its contribution is therefore not a new end-to-end association paradigm, but a re-engineering of two failure points inside the SORT/ByteTrack family: linear prediction and rigid overlap-based matching.

This distinction is important for comparative reading. Deep SORT uses memory as appearance retention, MeMOT uses memory as a large spatio-temporal latent store for end-to-end query decoding, and MeMoSORT uses memory as history-conditioned Bayesian correction plus motion-conditioned geometric association. The three systems all address identity continuity, but they do so at different points in the tracking pipeline and with different commitments regarding explicit matching, detector coupling, and model structure.

7. Limitations, scope, and typical use cases

The paper’s strengths are matched by several limitations that are either explicit or readily implied by the available technical description (Wang et al., 13 Aug 2025). MeMoSORT still depends on the detector and on dataset-calibrated thresholds, including confidence splitting and motion-adaptation cutoffs. Mo-IoU is strongly motivated by human tracking, particularly the usefulness of height similarity, so its transferability to object classes without stable or discriminative vertical extent is uncertain. The available text also leaves important implementation details unspecified: explicit loss functions for MeKF, architecture sizes, target definitions, sequence sampling details, and complete track-management logic are not provided.

The method also pays a speed cost for added modules. Although the system remains real-time in the reported experiments, throughput falls substantially when MeKF, Mo-IoU, and especially ReID are enabled. A further practical point is that MeMoSORT is not intended as an end-to-end jointly optimized detector-tracker; detector, ReID, and tracking components remain modular. This improves drop-in compatibility with existing tracking-by-detection frameworks, but it also means that global co-optimization is outside the method’s scope.

Within those limits, the intended operating regime is clear. MeMoSORT is most suitable for online tracking of people or similar objects in scenes with nonlinear motion, frequent occlusion, and weak appearance distinctiveness. Its central claim is that these conditions expose the insufficiency of the standard Kalman-filter-plus-IoU recipe, and that the combination of residual memory-assisted filtering and motion-adaptive association can repair those weaknesses without abandoning the efficiency of the SORT/ByteTrack paradigm (Wang et al., 13 Aug 2025).