Deep Learning in Video Multi-Object Tracking: A Survey (1907.12740v4)

Abstract: The problem of Multiple Object Tracking (MOT) consists in following the trajectory of different objects in a sequence, usually a video. In recent years, with the rise of Deep Learning, the algorithms that provide a solution to this problem have benefited from the representational power of deep models. This paper provides a comprehensive survey on works that employ Deep Learning models to solve the task of MOT on single-camera videos. Four main steps in MOT algorithms are identified, and an in-depth review of how Deep Learning was employed in each one of these stages is presented. A complete experimental comparison of the presented works on the three MOTChallenge datasets is also provided, identifying a number of similarities among the top-performing methods and presenting some possible future research directions.

• The paper thoroughly reviews how deep learning enhances video multi-object tracking, examining its impact on detection, feature extraction, affinity computation, and association.
• It highlights the use of CNNs and recurrent networks like LSTMs to reduce false negatives and improve motion prediction in challenging scenarios.
• The survey underscores emerging trends such as real-time processing, cross-scenario generalization, and improved association methods to minimize ID switches.

Deep Learning in Video Multi-Object Tracking: A Survey

The survey paper titled "Deep Learning in Video Multi-Object Tracking: A Survey" offers an extensive review of how Deep Learning (DL) approaches have transformed the field of Multiple Object Tracking (MOT) on single-camera video settings. This document meticulously breaks down the application of deep learning techniques across the four primary stages of the MOT pipeline and critiques the potential of DL in enhancing these stages.

Key Insights and Methodological Approaches

The MOT process is traditionally segmented into four steps: detection, feature extraction/motion prediction, affinity computation, and association. Each of these steps has benefited from the advent of DL, yet they present unique challenges and opportunities for improvement that researchers are actively exploring.

1. Detection:
   • Algorithms utilizing advanced versions of CNN-based detectors such as Faster R-CNN and SSD have demonstrated marked improvements in performance metrics, particularly in reducing false negatives—one of the most significant challenges in MOT. While private detections have generally outperformed public detections, suggesting a high potential for custom-trained models, the computational demand of these techniques remains a constraint for real-time applications.
2. Feature Extraction and Motion Prediction:
   • The power of deep networks, particularly CNNs, is their ability to extract robust appearance features, crucial for discerning objects in busy scenes. Architectures like Siamese networks and LSTMs are frequently employed to derive temporal features, enhancing the system's ability to predict object movement and interaction within the video context.
3. Affinity Computation:
   • The use of DL to compute affinities has gained traction with networks trained to output similarity scores directly, bypassing traditional approaches that rely on handcrafted distance measures. This is exemplified by the application of LSTMs and Siamese networks which have started to improve affinity calculations significantly by leveraging complex feature interactions.
4. Association:
   • Although DL's application in the association phase is less widespread, there are promising methods utilizing neural networks to streamline the ID assignment process. This represents a ripe area for further exploration, especially in reducing ID switches and enhancing the management of track initiations and terminations.

Benchmarking and Evaluations

The survey extensively covers experimental results on the MOTChallenge datasets (MOT15, MOT16, MOT17), underscoring that DL-based approaches, while computationally intensive, hold a significant edge in challenging scenarios. The paper emphasizes the importance of reducing false negatives through better detection and feature extraction strategies while also noting the potential trade-offs between batch and online processing methods.

Implications and Future Directions

The review identifies several emerging trends and potential research directions:

• Detection Robustness: Improving detection frameworks to handle variability in occlusion, background complexity, and object densities remains pivotal.
• Real-time Processing: Bridging the gap between high accuracy and real-time performance through optimization and hybrid methods is a continuous challenge.
• Cross-Scenario Generalization: Adapting current methodologies to generalized scenarios outside pedestrian tracking, such as in custom object classes, poses an interesting challenge.
• Enhanced Association Algorithms: DL approaches that explicitly address association process complex dynamics could significantly reduce ID switches, a persistent issue in dynamic scenes.

Conclusion

Ultimately, this survey consolidates a substantial body of knowledge on the integration of DL in video MOT, offering a framework for understanding how deep models can address existing challenges in object tracking. While DL has already made significant strides in tackling central issues, further innovation is essential to advance the field towards more adaptive, accurate, and efficient MOT systems.