ByteTrack: Efficient Object Tracking

• ByteTrack is a multiple object tracking algorithm that employs a cascaded association strategy to utilize both high and low confidence detection boxes.
• It features a dual-phase matching process using Kalman filtering and the Hungarian algorithm, achieving state-of-the-art metrics such as an 80.3 MOTA on MOT17.
• The method is applicable across diverse domains—from pedestrian tracking to precision livestock monitoring—offering robust performance in occluded and cluttered environments.

ByteTrack is a multiple object tracking (MOT) algorithm that operates within the tracking-by-detection paradigm, distinguished by its generic and efficient cascaded association strategy designed to exploit all available detection boxes—including those with low confidence scores. It achieved state-of-the-art benchmarks on MOT17 and MOT20 at real-time speeds and has since become a baseline for a range of MOT applications, from pedestrian tracking to high-throughput phenotyping and precision livestock monitoring. The method’s influence is rooted in its capacity to reduce missed detections and fragmentation by associating every detection box, not just the high-scoring ones, with a simple and computationally lightweight methodology.

1. Algorithmic Framework and Association Strategy

ByteTrack’s central innovation is the two-stage matching procedure that divides detection boxes into high- and low-confidence groups using a score threshold, $\tau$. In the first phase, existing tracklets are associated with high-confidence detections using motion-based cues (typically Intersection over Union, IoU, after Kalman filter prediction). This association employs the Hungarian algorithm to solve the linear assignment problem. The cost for associating predicted tracklet $T_j$ with detection $d_i$ is given by

$C(i, j) = -\text{IoU}(T_j, d_i)$

where

$$\text{IoU}(B_1, B_2) = \frac{|B_1 \cap B_2|}{|B_1 \cup B_2|}$$

Any unmatched tracklets from the first phase are then processed against the low-confidence detections in a second phase, again using IoU. This dual-phase matching ensures recovery of true objects that may have low detection scores due to occlusion, blur, or other environmental noise, substantially reducing missed objects and fragments in resulting trajectories.

2. Technical Details, Implementation, and Metrics

The ByteTrack pipeline consists of the following stages per frame:

1. Detection: The detector outputs bounding boxes and associated scores.
2. Splitting: Boxes are separated into $D_\text{high}$ and $D_\text{low}$ where

$$D_\text{high} = \{d: \text{score}(d) \geq \tau\},\qquad D_\text{low} = D \setminus D_\text{high}$$

1. Prediction: Tracklet positions are updated via Kalman filtering:

$x_t = F x_{t-1} + w_t$

where $F$ is the state transition matrix.

1. First Phase Association: Tracklets and $D_\text{high}$ matched via the Hungarian algorithm, using negative IoU as the cost.
2. Second Phase Association: Remaining unmatched tracklets are matched to $D_\text{low}$ using the same matching logic.
3. Update: Tracklets matched in each association phase are updated; unmatched low-score detections are discarded.

A representative interpolation strategy for occluded targets uses a linear interpolation formula for bounding boxes:

$$B_t = B_{t_1} + (B_{t_2} - B_{t_1}) \cdot \frac{t - t_1}{t_2 - t_1}$$

Key performance metrics assessed in benchmark evaluations include Multi-Object Tracking Accuracy (MOTA), Identification F1 score (IDF1), and Higher Order Tracking Accuracy (HOTA). ByteTrack achieved 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on MOT17, operating at 30 FPS on a V100 GPU (Zhang et al., 2021).

3. Impact, Benchmark Results, and Comparisons

Extensive experiments on MOT17, MOT20, HiEve, and BDD100K illustrate robust performance, with ByteTrack consistently outperforming previous baselines especially in crowded scenes. When ByteTrack’s association was applied to 9 state-of-the-art trackers, improvements in IDF1 of 1–10 points were reported (Zhang et al., 2021).

Compared to methods relying strictly on high-score detections, ByteTrack’s approach reduces the fragmentation of tracks and the loss of objects, yielding lower identity switches and higher overall accuracy, particularly in challenging conditions such as occlusion, motion blur, and cluttered backgrounds.

However, in settings with complex nonlinear or erratic motion (e.g., DanceTrack or fast-moving tiny objects), ByteTrack’s reliance on a linear Kalman filter and geometric-only association can lead to suboptimal performance. Deep learning-based association methods or explicit motion modeling methods (e.g., DeepOCSORT, MOTR) sometimes outperform ByteTrack in such domains (Zeng et al., 2021, Singh et al., 22 Sep 2025).

4. Extensions, Variants, and Practical Adaptations

Following the success of ByteTrack, several adaptations and improvements have been proposed:

• ByteTrackV2 extends the hierarchical association to both 2D and 3D tracking, and introduces a complementary motion prediction strategy where Kalman-predicted velocities are combined with detection-derived velocities for improved handling of abrupt motion in 3D space (Zhang et al., 2023).
• Adaptive Thresholding replaces the static threshold $\tau$ with a data-driven method that dynamically chooses the cutoff via the steepest descent in the sorted list of detection scores, eliminating the need for per-dataset tuning. Experiments show similar HOTA, MOTA, and IDF1 performance as the baseline ByteTrack with negligible computational cost (Ma et al., 2023).
• Resource-Constrained Adaptations: Multi-Resolution Rescored ByteTrack (MR2-ByteTrack) combines low- and high-resolution detection with probabilistic rescoring, significantly reducing latency and compute load on microcontrollers while maintaining or improving tracking accuracy (Bompani et al., 17 Apr 2024).

5. Applications and Empirical Evaluations

ByteTrack’s generic, detector-agnostic pipeline has led to adoption in diverse application domains:

• Retail Analytics: Used with YOLOv8 as the detection front-end, ByteTrack produces robust customer tracking at 17 FPS, facilitating accurate visitor counting and behavioral heat maps (Hossam et al., 24 Feb 2024).
• Biomedical Imaging: In ProGroTrack, ByteTrack links detections of protein fibers from YOLO models, supporting fine-resolution dynamic analysis in intracellular studies (Chan et al., 2023).
• Agricultural Phenotyping: In high-throughput seed kernel counting, ByteTrack yields counting accuracy exceeding 96% for soy and 92% for wheat at 120 fps, outperforming or matching StrongSORT (Margapuri et al., 2023).
• Precision Livestock Farming: ByteTrack’s superior IDF1 and MOTA scores (IDF1 ≈ 0.79, MOTA ≈ 0.73) make it more reliable than traditional tools such as DeepLabCut or idTracker for long-term multi-animal tracking in pig pens, supporting downstream behavior analysis and welfare assessment (Bibinbe et al., 15 Sep 2025).

ByteTrack is routinely combined with deep learning detectors (YOLOv8/v10, YOLOX) and big data pipelines (Apache Kafka, Apache Spark) to maintain real-time operation in high-throughput or distributed systems (e.g., RAMOTS for UAV-driven aerial MOT, achieving HOTA of 48.14 at 28 FPS) (Do et al., 6 Feb 2025).

6. Limitations, Robustness, and Ongoing Challenges

ByteTrack’s reliance on geometric association and a constant velocity Kalman filter leads to limitations, particularly under:

• Nonlinear/erratic motion: Significant tracking drift and high ADE (Average Displacement Error) compared to appearance-augmented or advanced motion modeling approaches. In racquetball scenarios, ByteTrack’s ADE (114.3 px) is several times higher than that of DeepOCSORT, despite faster processing (Singh et al., 22 Sep 2025).
• Adversarial Perturbations: ByteTrack is vulnerable to adversarial attacks that perturb the detection output (as with the Tracklet-Switch Adversarial Attack, “TraSw”), leading to high attack success rates and persistent identity switches (Lin et al., 2021).
• Occlusion/Fragmentation Over Long Horizons: Despite its robustness in many scenarios, over long-term deployments (e.g., 10-minute livestock tracking), performance can degrade due to accumulated identity switches and missed associations, although integration with probabilistic frameworks such as HMM smoothing markedly improves stability (micro-F1 rises from ≈0.48–0.50 to ≈0.67) (Bibinbe et al., 12 Sep 2025, Bibinbe et al., 15 Sep 2025).

Efforts to address these deficits include integrating adaptive Kalman filters (variable speed or ego-motion-aware), learning-based similarity or affinity functions, modular pseudo-3D cues (Limanta et al., 26 Sep 2024), and temporal consistency modules.

7. Legacy, Influence, and Future Research Directions

ByteTrack has catalyzed both research and practical advances in the MOT field:

• It established a strong, efficient, and extensible baseline for tracking-by-detection, inspiring hierarchical, modular, and hybrid pipelines now prominent in real-time and large-scale deployments (Adžemović, 16 Jun 2025).
• Its association strategy is now frequently adopted or adapted for both 2D and 3D domains, with many subsequent trackers (e.g., ByteTrackV2, SMILEtrack, HIT) either extending or directly benchmarking against ByteTrack (Wang et al., 2022, Zhang et al., 2023, Du et al., 19 Jun 2024).
• Its limitations—in robustness to motion complexity, appearance ambiguity, and adversarial manipulation—serve as reference points for future end-to-end, attention-based, and multi-modal tracking architectures.

In sum, ByteTrack remains technically significant both as a benchmark and as a practical MOT solution, owing to its universal association strategy, robust benchmark performance, and broad cross-domain applicability.