Deep Network Flow for Multi-Object Tracking (1706.08482v1)

Abstract: Data association problems are an important component of many computer vision applications, with multi-object tracking being one of the most prominent examples. A typical approach to data association involves finding a graph matching or network flow that minimizes a sum of pairwise association costs, which are often either hand-crafted or learned as linear functions of fixed features. In this work, we demonstrate that it is possible to learn features for network-flow-based data association via backpropagation, by expressing the optimum of a smoothed network flow problem as a differentiable function of the pairwise association costs. We apply this approach to multi-object tracking with a network flow formulation. Our experiments demonstrate that we are able to successfully learn all cost functions for the association problem in an end-to-end fashion, which outperform hand-crafted costs in all settings. The integration and combination of various sources of inputs becomes easy and the cost functions can be learned entirely from data, alleviating tedious hand-designing of costs.

• The paper introduces a method that learns cost functions in a network flow framework for multi-object tracking using end-to-end backpropagation.
• It employs bi-level optimization and smoothing techniques to integrate diverse features and enhance data association accuracy.
• Empirical evaluations on KITTI and MOT16 benchmarks show improved performance with fewer identity switches and reduced trajectory fragmentations.

Deep Network Flow for Multi-Object Tracking

The paper presents an innovative approach to multi-object tracking (MOT) by introducing a method to learn cost functions for network-flow-based data association using backpropagation. The authors address the complexity of hand-crafting cost functions for data association by formulating them as learnable parameters within the network-flow problem, thus allowing for a more adaptive and potentially more accurate tracking solution in computer vision applications.

Core Contributions and Methods

The research is anchored on a network flow framework that addresses the data association problem inherent in MOT. Traditionally, data association is resolved through graph matching approaches or network flows, which utilize cost functions typically designed manually or as linear functions based on fixed features. This paper extends the existing paradigms by proposing a mechanism to learn these cost functions in an end-to-end manner. By employing bi-level optimization within a smoothed linear program (LP), the paper enables backpropagation through the solution of the network flow problem.

Key advancements include:

1. End-to-End Learning: Instead of relying on handcrafted functions, the proposed model learns all cost function parameters directly from data. This approach alleviates the burdens of manual cost tuning and allows the integration of diverse input types including bounding boxes and various image features.
2. Scalability and Flexibility: Their framework is scalable across different types of association problems and can work with non-linear differentiable functions such as neural networks.
3. Smoothing Techniques: The authors utilize smoothing methods to approximate the LP constraints, making it easier to incorporate these within deep learning architectures.

Empirical Evaluation

The experiments highlight the effectiveness of the proposed method on popular MOT benchmarks like KITTI-Tracking and MOT16. Several models are compared, showcasing the superior performance of learned cost functions over handcrafted ones. For instance, the proposed approach achieved better MOTA scores, fewer identity switches, and fewer trajectory fragmentations than traditional hand-crafted approaches. These results demonstrate the potential of the learned models to adaptively learn the dynamics and interactions within video sequences more effectively than static, predefined models.

Implications and Future Work

This research offers substantial implications for the future of MOT and related computer vision fields. By simplifying the incorporation of new features into cost functions, the model not only improves tracking performance but also opens doors for integrating advanced features like deep learning-driven object detection and appearance models.

Looking forward, several paths for further investigations are suggested:

• Integration with Detectors: A comprehensive integration of object detectors within this framework could streamline the tracking process even further by harmonizing detection and tracking phases.
• Complex Network Flow Models: Exploring more elaborate network flow graphs, including trajectory interactions and dependencies, could further enhance prediction accuracy.
• Extensions to Max-Flow Problems: Adapting the techniques for max-flow problems indicates a possible extension to other optimization challenges in image processing and beyond.

In conclusion, the paper lays foundational work for a paradigm shift in multi-object tracking by demonstrating that learning network flow costs from data is not only feasible but advantageous, thus potentially reshaping approaches to data association in complex, dynamic systems.