Simple Online and Realtime Tracking with a Deep Association Metric (1703.07402v1)

Abstract: Simple Online and Realtime Tracking (SORT) is a pragmatic approach to multiple object tracking with a focus on simple, effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer periods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a large-scale person re-identification dataset. During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experimental evaluation shows that our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates.

• The paper introduces Deep SORT, an enhanced version of SORT that integrates deep appearance metrics to significantly reduce identity switches.
• It reformulates data association using a weighted combination of Mahalanobis and cosine distance metrics derived from a pre-trained CNN.
• Experimental results on the MOT16 dataset show improved tracking accuracy with a 45% reduction in identity switches while maintaining real-time performance.

Simple Online and Real-time Tracking with a Deep Association Metric

The paper "Simple Online and Real-time Tracking with a Deep Association Metric" by Nicolai Wojke, Alex Bewley, and Dietrich Paulus presents an enhancement to the Simple Online and Realtime Tracking (SORT) algorithm, with an emphasis on integrating appearance information to improve tracking performance.

Introduction and Background

In the domain of multiple object tracking (MOT), tracking-by-detection has emerged as the dominant paradigm. Traditional methods such as Multiple Hypothesis Tracking (MHT) and the Joint Probabilistic Data Association Filter (JPDAF) have been commonly used for this purpose. While these methods offer robustness and accuracy, they are often computationally expensive and complex to implement, especially in real-time scenarios.

SORT stands out by utilizing a simple yet effective approach, employing Kalman filtering in image space and frame-by-frame data association using the Hungarian method. However, SORT has a notable deficiency: a relatively high number of identity switches, especially in situations involving occlusions. The paper addresses this limitation by integrating a deep appearance descriptor to enhance the data association process.

Core Contributions

The paper introduces a modified version of SORT that incorporates a deep association metric, leveraging convolutional neural networks (CNN) to improve the robustness of the tracking algorithm. Key aspects include:

1. Integration of Appearance Information: By embedding appearance information into the data association process, the modified SORT algorithm (termed Deep SORT) reduces identity switches significantly. This is achieved using a pre-trained CNN that provides a discriminative feature embedding for appearance descriptors.
2. Assignment Problem Reformulation: The data association is performed using a weighted combination of Mahalanobis distance and a cosine distance metric based on appearance descriptors. This composite metric allows for more accurate tracking through occlusions.
3. Matching Cascade: To handle longer-term occlusions, the authors propose a matching cascade that prioritizes tracks based on their recency of observation. This approach mitigates the issue of track fragmentation while maintaining computational efficiency.

Experimental Results

The authors conducted extensive experiments on the MOT16 benchmark dataset, which consists of diverse and challenging sequences. Key findings include:

• Reduction in Identity Switches: Deep SORT achieves a 45% reduction in identity switches (from 1423 to 781) compared to the original SORT algorithm, highlighting the effectiveness of incorporating appearance information.
• Overall Performance: Deep SORT not only reduces identity switches but also shows improvements in the number of mostly tracked (MT) objects and a decrease in mostly lost (ML) objects.
• Competitive Real-time Performance: Despite the enhancements, the modified algorithm maintains a high frame rate of approximately 20 Hz, making it suitable for real-time applications.

The performance metrics used for evaluation include Multi-object Tracking Accuracy (MOTA), Multi-object Tracking Precision (MOTP), and various track-related statistics such as identity switches (ID) and track fragmentations (FM). Deep SORT demonstrates a balanced enhancement in these metrics, positioning itself as a strong competitor to both online and batch processing methods in the MOT domain.

Implications and Future Work

The paper's contributions have significant implications for multiple object tracking in real-time applications. The use of a deep association metric not only improves tracking accuracy but also maintains the simplicity and efficiency of the original SORT framework. This makes Deep SORT particularly suitable for real-time applications such as surveillance, autonomous driving, and robotics.

Future developments could explore the integration of more sophisticated deep learning models and metric learning techniques to further enhance the tracking performance. Additionally, examining the applicability of this approach to different types of objects beyond pedestrians could provide broader insights into the versatility of the proposed method.

Conclusion

The modification of SORT by incorporating a deep association metric represents an important step in improving real-time multiple object tracking. By addressing the challenges posed by occlusions and identity switches, this paper contributes to the robustness and practicality of tracking algorithms in dynamic and complex environments. The combination of simplicity, efficiency, and improved accuracy makes Deep SORT a valuable tool in the field of computer vision.