BoT-SORT-ReID: Online Pedestrian Tracker

• BoT-SORT-ReID is a modular online multi-object tracker that fuses motion cues with FastReID appearance embeddings using a gate-then-min association strategy.
• It refines state estimation through a Kalman filter with direct width-height prediction and compensates camera motion using affine transforms for robust tracking.
• Empirical evaluations on MOT17 and MOT20 benchmarks demonstrate top performance with metrics of 80.5 MOTA, 80.2 IDF1, and 65.0 HOTA while reducing identity switches.

BoT-SORT-ReID is an online multi-object tracker for multi-pedestrian tracking in the tracking-by-detection setting. It extends BoT-SORT by adding a person re-identification branch to a pipeline that already combines an updated Kalman state, camera-motion compensation, and ByteTrack-style two-stage association. Its defining design choice is not merely the presence of appearance embeddings, but the specific way motion and appearance are fused: implausible pairs are rejected by IoU and embedding gates, and the final first-stage assignment cost is taken as the minimum of IoU distance and a masked cosine distance, rather than a conventional weighted sum. In the original report, BoT-SORT-ReID ranked first on the MOTChallenge benchmarks at the time of publication, achieving 80.5 MOTA, 80.2 IDF1, and 65.0 HOTA on MOT17 (Aharon et al., 2022).

1. Historical position and design objective

BoT-SORT-ReID emerged as a response to a persistent trade-off in online MOT systems: motion-dominant trackers often preserve MOTA but incur more identity switches, whereas appearance-heavy trackers can improve IDF1 while degrading localization-sensitive performance. The method is presented as a robust tracker that combines motion information, appearance information, camera-motion compensation, and a more accurate Kalman filter state vector within a single online framework. It builds on BoT-SORT, which itself improves ByteTrack, and adds a ReID module intended to resolve the MOTA–IDF1 trade-off rather than merely append an embedding stage to an existing association routine (Aharon et al., 2022).

Relative to contemporaries, the method differs from DeepSORT in two fundamental ways. First, it replaces the DeepSORT-style state parameterization based on aspect ratio and height with a state that estimates width and height directly. Second, it abandons the common weighted fusion of motion and appearance in favor of a gate-then-min strategy. Relative to ByteTrack, it retains the two-stage association over high- and low-confidence detections, but augments that regime with explicit camera-motion compensation and ReID-based disambiguation. The appearance-free BoT-SORT and the appearance-enabled BoT-SORT-ReID therefore share most of their motion machinery; the distinction lies in whether FastReID-derived embeddings are injected into the first association stage (Aharon et al., 2022).

The method is best understood as a modular MOT architecture with four coupled subsystems: a detector, a compensated Kalman predictor, a ReID encoder with track-level feature aggregation, and a staged assignment procedure. Later work has treated that modularity as the main reason BoT-SORT-ReID became a reference baseline: subsequent papers replace the appearance branch, change when embeddings are computed, or alter the motion–appearance fusion while leaving the broader BoT-SORT tracking skeleton intact (Alikhanov et al., 2024).

2. State estimation and camera-motion compensation

A central technical departure from DeepSORT is the Kalman state itself. BoT-SORT-ReID uses the constant-velocity state

$$x_k = [x_c(k), y_c(k), w(k), h(k), \dot{x}_c(k), \dot{y}_c(k), \dot{w}(k), \dot{h}(k)]^\top$$

with measurement

$$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$

The choice to estimate width and height directly, rather than aspect ratio and height, is reported to yield tighter localization and better overlap for association. Prediction and update follow the standard linear-Gaussian Kalman recursion with constant-velocity transition matrix $F$ and observation matrix $H$ (Aharon et al., 2022).

The process and measurement covariances are scaled by predicted box size. In the reported implementation, the noise factors are $\sigma_p = 0.05$, $\sigma_v = 0.00625$, and $\sigma_m = 0.05$ for a 30 FPS regime. This size-aware parameterization is part of the method’s attempt to preserve accurate IoU geometry under scale changes, since poor width-height prediction propagates directly into worse association even when appearance cues are available (Aharon et al., 2022).

Camera-motion compensation is the other major motion-side innovation. Between adjacent frames, BoT-SORT-ReID estimates a global affine transform using Shi–Tomasi keypoints, Pyramidal Lucas–Kanade optical flow, and RANSAC. If

$A_{k-1}^k = [M \mid T]$

denotes the frame-to-frame affine warp, then the predicted state and covariance are transformed before association:

$$\hat{x}_{k|k-1}^\prime = \tilde{M}_{k-1}^k \hat{x}_{k|k-1} + \tilde{T}_{k-1}^k,$$

$$P_{k|k-1}^\prime = \tilde{M}_{k-1}^k P_{k|k-1} (\tilde{M}_{k-1}^k)^\top.$$

The compensated prior is then passed into the usual Kalman update. This design explicitly treats inter-frame camera motion as a nuisance variable that should be removed before geometric matching, rather than absorbed indirectly by larger motion uncertainty (Aharon et al., 2022).

The significance of this choice was supported by ablations on MOT17 validation, where camera-motion compensation and the updated state vector increased HOTA and IDF1. The paper also used a cumulative MOTA diagnostic, cMOTA, to show that compensation recovers performance in failure segments such as camera rotations. A plausible implication is that BoT-SORT-ReID’s appearance branch is most effective when geometric prediction has already been stabilized, because the ReID module is then reserved for genuinely ambiguous associations rather than systematic background-induced misalignment (Aharon et al., 2022).

3. Appearance modeling and association procedure

The ReID branch in BoT-SORT-ReID uses FastReID’s strong baseline with a ResNeSt-50 backbone. It is trained for 60 epochs with FastReID’s default recipe on the first half of MOT17 or MOT20 training sequences and validated on the second half under the private detection protocol. At inference, embeddings are extracted only for high-confidence detections, which reduces both noise and computation. Per-track appearance is maintained by exponential moving average,

$$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$0

with $$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$1, where $$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$2 is the embedding of the matched detection at frame $$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$3 (Aharon et al., 2022).

The first association stage operates on high-confidence detections. Detection scores are split at threshold $$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$4, and unmatched high-confidence detections can initialize new tracks if their score exceeds $$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$5. ReID extraction is performed only on the high-confidence set. The second stage then matches remaining tracks to low-confidence detections using IoU-only assignment, following the ByteTrack principle of recovering recall from low-score boxes (Aharon et al., 2022).

The appearance cue is encoded through cosine distance, while geometric consistency is encoded through IoU distance. Before assignment, BoT-SORT-ReID applies gates: an IoU proximity threshold $$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$6, an appearance threshold $$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$7, and an implementation-level minimum similarity rejection such as similarity $$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$8. The cosine distance is then masked as

$$z_k = [z_{x_c}(k), z_{y_c}(k), z_w(k), z_h(k)]^\top.$$9

and the final first-stage cost is

$F$0

This “mask-and-min” rule is the method’s main association signature (Aharon et al., 2022).

The rationale for the minimum operator is that the stronger cue should dominate only after implausible pairs have already been removed. The paper explicitly contrasts this with the weighted sum

$F$1

which had been common in earlier trackers. In BoT-SORT-ReID, appearance is not allowed to override geometry indiscriminately; it resolves ambiguous motion matches only within a filtered candidate set. This is why the method can be described more precisely as gated appearance-assisted association rather than appearance-fused matching in the generic DeepSORT sense (Aharon et al., 2022).

Component	BoT-SORT-ReID specification	Role
Detection split	$F$2	High/low confidence partition
New-track threshold	$F$3	Track initialization
IoU gate	$F$4	Geometric plausibility
Appearance gate	$F$5	Embedding plausibility
Appearance update	EMA, $F$6	Track feature smoothing
Lost-track TTL	30 frames	Short-term re-association

Track management follows the usual online regime: matched tracks update state and covariance; matched high-confidence detections update the appearance EMA; unmatched tracks become lost and can be retained for up to 30 frames; and optional linear interpolation with maximum gap 20 is applied offline for evaluation fairness. This keeps the tracker strictly online during association while still permitting a limited post-processing step in benchmark evaluation (Aharon et al., 2022).

4. Evaluation and comparative interpretation

On MOT17 under the private-detector protocol, BoT-SORT-ReID achieved 80.5 MOTA, 80.2 IDF1, and 65.0 HOTA, and was reported as ranking first on all three main MOT metrics at the time of publication. The paper also states that it was the first tracker to surpass 80 IDF1 on MOT17. On MOT20, it likewise ranked first in MOTA, IDF1, and HOTA, with the excerpt noting significant IDF1 and HOTA improvements while preserving MOTA in crowded sequences, although exact MOT20 values are not reproduced in the provided text (Aharon et al., 2022).

The ablation narrative is as important as the headline metrics. The reported validation analysis found that IoU-only association can outperform naïve cosine-based fusion at high frame rates, which directly motivates the gate-then-min formulation. The proposed fusion then yields the highest MOTA, IDF1, and HOTA among the tested strategies. This is a useful corrective to the common misconception that adding a stronger ReID encoder alone is sufficient: BoT-SORT-ReID attributes a large share of its gains to the interaction between improved geometry, compensation, gating, and conservative appearance use, not to appearance descriptiveness in isolation (Aharon et al., 2022).

Later work reinforced the idea that BoT-SORT-ReID should be evaluated as part of a full pipeline rather than as an isolated association module. LITE, which replaces the external ReID encoder with detector-backbone descriptors, reported that under a holistic detector-plus-tracker evaluation on MOT17 with YOLOv8m, BoTSORT reached HOTA 40.9, IDF1 46.0, MOTA 41.1, and 19.1 FPS, while LITE:BoTSORT reached HOTA 40.8, IDF1 45.9, MOTA 41.1, and 30.7 FPS; on MOT20 the corresponding HOTA/FPS values were 20.8/14.3 for BoTSORT and 21.1/24.2 for LITE:BoTSORT. These numbers are not a remeasurement of the original MOTChallenge protocol, but they illustrate how the ReID branch can dominate end-to-end throughput when detector time is included (Alikhanov et al., 2024).

5. Later reinterpretations of the ReID branch

A substantial later literature has treated BoT-SORT-ReID less as a fixed algorithm than as a scaffold whose appearance subsystem can be swapped or scheduled. One line of work focuses on eliminating or reducing external ReID inference. LITE extracts descriptors directly from a detector backbone, specifically from the first YOLOv8m convolutional feature map with $F$7 channels at half spatial resolution. After non-maximum suppression, a detection box is mapped into feature-map coordinates, cropped, globally average pooled to $F$8, and L2-normalized:

$F$9

The resulting appearance distance is $H$0, optionally fused with IoU as

$H$1

In BoT-SORT terms, the proposal is to remove crop-resize and external ReID inference while keeping the Kalman predictor, feature-buffer logic, cascade, Hungarian assignment, and optional camera-motion compensation unchanged (Alikhanov et al., 2024).

A second line of work asks not how to replace ReID, but when to compute it. A selective extraction mechanism was proposed for StrongSORT and Deep OC-SORT and explicitly described for porting to BoT-SORT. For each detection $H$2, candidate confirmed tracks are defined by

$H$3

If $H$4, aspect-ratio similarity

$H$5

and its IoU-conditioned weight

$H$6

are used to decide whether the detection is “non-risky.” For non-risky cases, the method copies the candidate track’s feature into the detection instead of extracting a new embedding, so the appearance cost for that candidate becomes zero without disrupting the matching stage. A decay-aware EMA then updates track features when fresh embeddings are skipped. This suggests a BoT-SORT-compatible scheduler for ReID extraction under embedded or latency-constrained conditions (Bayar et al., 2024).

A third line treats the appearance encoder itself as replaceable. SimpleReID trains an unsupervised ReID classifier from pseudo-labels generated by SORT on unlabeled videos, then discards the classifier head at inference and uses the embedding as a drop-in descriptor. The paper reports that SimpleReID recovered supervised ReID performance inside Tracktor++ and DeepSORT and improved a CenterTrack-based system on MOT16/17. The accompanying technical brief describes direct substitution into BoT-SORT-style pipelines while keeping Kalman prediction, camera-motion compensation, Hungarian assignment, and cosine-based association unchanged. This reframes the ReID module as a trainable but supervision-optional component rather than an intrinsically supervised dependency (Karthik et al., 2020).

Additional replacements target the efficiency or robustness of the descriptor space. SCR and IntSCR compress real-valued ReID features into short sub-space codes and replace direct Euclidean comparisons with centroid look-up table summation, enabling counting-sort-based ranking; the provided integration guidance describes using these distances inside a BoT-SORT association matrix after normalization (Yin et al., 2022). SUReID instead addresses occlusion by introducing hierarchical token sparsification, non-parametric feature-alignment distillation, and noise occlusion augmentation in a ViT/DeiT-based ReID model; its integration note explicitly presents it as a drop-in BoT-SORT appearance module with cosine similarity on L2-normalized embeddings (Song et al., 2024).

6. Domain transfer, limitations, and continuing directions

BoT-SORT-ReID has also been adapted beyond pedestrian street scenes. In thermal infrared multi-UAV tracking, a 2025 system combined YOLOv12 with BoT-SORT and an SBS ResNet-50 ReID model trained with CircleLoss. For the Anti-UAV Track 3 benchmark, the reported best configuration was YOLOv12n at image size 1600 with BoT-SORT-SBS-S50, CircleLoss, AdamW, ReID training for 17 epochs, and track buffer 60, yielding 0.760874 MOTA. The same study attributes much of the improvement to higher detector resolution and small-object handling, but still retains appearance-first then IoU association, motion gating, and global motion compensation in the BoT-SORT style (Chen, 21 Mar 2025).

In maritime MOT, DMSORT positions itself directly against BoT-SORT by replacing the appearance branch with a lightweight transformer-based extractor, Li-TAE, and by changing fusion to a multiplicative clustering-optimized feature fusion:

$H$7

after nonlinear scaling and IoU gating of appearance distances. On the Singapore Maritime Dataset, the paper reports BoT-SORT to BoT-SORT⋆ gains of HOTA 72.13 to 75.47, IDF1 78.81 to 82.35, and MOTA 70.62 to 75.92 when integrating stronger detector and appearance components. This suggests that BoT-SORT-ReID remains a baseline against which newer domain-specific fusion strategies are judged, even when those strategies depart from its original mask-and-min rule (Tang et al., 6 Nov 2025).

The original limitations of BoT-SORT-ReID are relatively clear. Camera-motion compensation can fail when there are too few reliable background keypoints or when parallax is extreme, which can corrupt the compensated prediction. Separate appearance extraction adds computational overhead relative to appearance-free trackers, even though BoT-SORT-ReID mitigates this by encoding only high-confidence detections. ReID embeddings can also be noisy in crowded, highly occluded, or blurred views, and long-term occlusions or extreme scale changes remain difficult because the tracker is online and does not perform heavy global linking (Aharon et al., 2022).

Taken together, the subsequent literature does not displace BoT-SORT-ReID so much as anatomize it. LITE questions the necessity of a separate ReID branch; selective extraction questions whether every detection needs an embedding; SimpleReID questions whether supervised labels are necessary; SUReID questions whether the appearance encoder can be made both occlusion-robust and fast; and maritime or UAV adaptations question how much of the original fusion logic transfers to non-pedestrian domains (Alikhanov et al., 2024, Bayar et al., 2024, Karthik et al., 2020, Song et al., 2024, Chen, 21 Mar 2025, Tang et al., 6 Nov 2025). In that sense, BoT-SORT-ReID occupies a stable place in MOT research as both a high-performing tracker in its own right and a canonical decomposition of online tracking into detector, motion model, camera-motion compensation, appearance encoder, and association policy.