Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking (2203.14360v3)

Abstract: Kalman filter (KF) based methods for multi-object tracking (MOT) make an assumption that objects move linearly. While this assumption is acceptable for very short periods of occlusion, linear estimates of motion for prolonged time can be highly inaccurate. Moreover, when there is no measurement available to update Kalman filter parameters, the standard convention is to trust the priori state estimations for posteriori update. This leads to the accumulation of errors during a period of occlusion. The error causes significant motion direction variance in practice. In this work, we show that a basic Kalman filter can still obtain state-of-the-art tracking performance if proper care is taken to fix the noise accumulated during occlusion. Instead of relying only on the linear state estimate (i.e., estimation-centric approach), we use object observations (i.e., the measurements by object detector) to compute a virtual trajectory over the occlusion period to fix the error accumulation of filter parameters during the occlusion period. This allows more time steps to correct errors accumulated during occlusion. We name our method Observation-Centric SORT (OC-SORT). It remains Simple, Online, and Real-Time but improves robustness during occlusion and non-linear motion. Given off-the-shelf detections as input, OC-SORT runs at 700+ FPS on a single CPU. It achieves state-of-the-art on multiple datasets, including MOT17, MOT20, KITTI, head tracking, and especially DanceTrack where the object motion is highly non-linear. The code and models are available at \url{https://github.com/noahcao/OC_SORT}.

• The paper introduces an observation-centric update that recalibrates Kalman filter estimates to mitigate noise during occlusions.
• It employs a momentum-based data association method that integrates motion direction consistency to improve tracking accuracy.
• OC-SORT achieves superior performance across challenging datasets and runs at over 700 FPS, underscoring its real-time applicability.

An Expert Overview of Observation-Centric SORT for Multi-Object Tracking

The paper "Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking" introduces a novel methodology for improving multi-object tracking (MOT) by addressing inherent limitations in Kalman filter-based approaches, particularly under occlusion and non-linear motion scenarios. This work builds upon the Simple Online and Realtime Tracking (SORT) framework, offering enhancements that render it more resilient and effective for diverse practical applications.

Research Context and Problem Statement

Kalman filter (KF)-based methods are prevalent in MOT due to their efficiency in handling linear motion within short time intervals. However, they inherently assume constant linear motion and become susceptible to inaccuracies during extended periods of occlusion or when objects exhibit non-linear motion patterns. These inaccuracies primarily arise due to error accumulation from relying solely on state estimations when observations are unavailable (dummy updates), causing significant complications in tracking robustness. The presented work identifies and addresses three main limitations of the traditional SORT: sensitivity to state noise, temporal error magnification, and an estimation-centric approach that overlooks the corrective potential in observations.

The Proposed Observation-Centric SORT (OC-SORT)

OC-SORT introduces a paradigm shift by adopting an "observation-centric" approach to mitigate error accumulation and improve tracking robustness. This approach incorporates two key innovations:

1. Observation-Centric Re-Update (ORU): This component activates when a previously untracked object is re-associated with an observation. ORU uses past observations to simulate a virtual trajectory, effectively re-updating the KF parameters to counteract noise accumulation during untracked intervals. This observation-centric correction proves crucial in recalibrating the KF parameters, ensuring that the tracker remains aligned with actual object motion trajectories.
2. Observation-Centric Momentum (OCM): This addition improves data association during tracking by integrating motion direction consistency into the cost matrix calculation. By using observations from non-consecutive frames, OCM reduces noise in direction estimation, thereby enhancing the reliability of the direction-based association in the presence of state noise.

Evaluation and Results

OC-SORT was rigorously evaluated against state-of-the-art methods on multiple datasets, including MOT17, MOT20, KITTI, DanceTrack, and CroHD, each presenting unique challenges with respect to occlusions and complex motion patterns. Notably, the OC-SORT method exhibited superior performance on the DanceTrack dataset, which is particularly challenging due to its non-linear motion scenarios. Across these datasets, OC-SORT consistently emerged as a top performer, validating its robustness and efficiency. Significantly, the method operates at an impressive 700+ FPS on a single CPU, underscoring its practicality for real-time applications.

Implications and Future Directions

The improvements demonstrated by OC-SORT have practical implications for a wide range of applications where robustness to occlusion and non-linear motion is critical, such as surveillance, autonomous driving, and sports analytics. The observation-centric approach provides a more resilient tracking system that can better manage adversarial conditions common in real-world scenarios.

The demonstrated success of OC-SORT opens several avenues for future exploration. There is potential to integrate this framework with deep learning-based feature extractors to further enhance detection and association accuracy. Additionally, further theoretical exploration and validation of trajectory hypothesis generation methods like Gaussian Processes could yield even more sophisticated interpolation strategies. These enhancements promise to advance the state of MOT systems, leading to broader adoption and application in increasingly complex environments.

In conclusion, the paper provides a comprehensive strategy to overcome traditional challenges in state estimation-based tracking systems. The observation-centric model not only elevates the performance of SORT but also sets a direction towards more adaptive and contextual MOT solutions.