Fast Online Object Tracking and Segmentation: A Unifying Approach (1812.05050v2)

Abstract: In this paper we illustrate how to perform both visual object tracking and semi-supervised video object segmentation, in real-time, with a single simple approach. Our method, dubbed SiamMask, improves the offline training procedure of popular fully-convolutional Siamese approaches for object tracking by augmenting their loss with a binary segmentation task. Once trained, SiamMask solely relies on a single bounding box initialisation and operates online, producing class-agnostic object segmentation masks and rotated bounding boxes at 55 frames per second. Despite its simplicity, versatility and fast speed, our strategy allows us to establish a new state of the art among real-time trackers on VOT-2018, while at the same time demonstrating competitive performance and the best speed for the semi-supervised video object segmentation task on DAVIS-2016 and DAVIS-2017. The project website is http://www.robots.ox.ac.uk/~qwang/SiamMask.

• The paper introduces SiamMask, a method that integrates online tracking and video segmentation into a unified multi-task framework.
• It leverages a ResNet-50 backbone with an added segmentation branch to achieve 55 fps and an EAO of 0.380 on VOT-2018.
• SiamMask simplifies real-time visual analysis by providing both bounding boxes and pixel-level segmentation, making it ideal for various applications.

Fast Online Object Tracking and Segmentation: A Unifying Approach

The paper "Fast Online Object Tracking and Segmentation: A Unifying Approach" by Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip H.S. Torr presents an effective method, SiamMask, to address both visual object tracking and semi-supervised video object segmentation (VOS). This model leverages the capabilities of fully-convolutional Siamese networks to achieve high performance in both domains without diverging from the requirements of real-time processing.

Methodology

SiamMask operates based on the premise of enhancing existing fully-convolutional Siamese networks, such as SiamFC and SiamRPN, by incorporating an additional segmentation branch. The key feature of this method is its capability to generate pixel-wise binary segmentation masks in conjunction with bounding box predictions. The model effectively unifies the tasks of object tracking and video object segmentation.

Network Architecture and Loss Functions

SiamMask employs a ResNet-50 backbone truncated before the final convolutional layer to preserve higher spatial resolution and extends it with a multi-task head comprising three branches: score, bounding box, and mask prediction. During training, the cross-entropy loss for classification scoring and smooth $L1$ loss for bounding box regression are incorporated alongside a binary logistic loss for segmentation, creating a comprehensive multi-task training objective. These components collectively ensure that the resultant model outputs a detailed representation of the target object as a segmentation mask, while still providing bounding boxes for compatibility with traditional tracking evaluations.

Experimental Evaluation

The efficacy of SiamMask was thoroughly evaluated across both tracking and segmentation benchmarks, showcasing its flexible application.

Visual Object Tracking

For visual object tracking, SiamMask was tested on VOT-2016 and VOT-2018 benchmarks using Expected Average Overlap (EAO) as the main metric. The results reveal that SiamMask establishes a new standard in real-time performance, achieving an EAO of 0.380 on VOT-2018, outperforming recent state-of-the-art methods like DaSiamRPN and SA_Siam_R, which are known for their high tracking accuracy. The model is capable of operating at 55 frames per second, thus meeting the requirements of real-time tracking applications.

Video Object Segmentation

For semi-supervised VOS, the evaluation spanned DAVIS-2016 and DAVIS-2017 benchmarks. SiamMask demonstrated competitive performance, achieving a $\mathcal{J}_{\mathcal{M}}$ index of 71.7% on DAVIS-2016 while running in real-time at 55 fps. Importantly, despite the increased complexity of generating segmentation masks, SiamMask's performance remained robust over time, indicating its suitability for long video sequences.

Implications and Future Work

SiamMask marks an important advancement in the field of visual object tracking and segmentation by effectively bridging the two tasks. The ability to produce segmentation masks online and in real-time introduces new possibilities for applications that require nuanced object delineation, such as autonomous driving and real-time video editing.

From a theoretical perspective, the use of multi-task learning within a fully-convolutional Siamese framework sets a precedent for future studies to explore more sophisticated integration techniques. The implementation of FPN-like architectures for refining segmentation masks demonstrates the potential of sophisticated feature merging strategies, which can be further enhanced with additional architectural innovations.

Practically, SiamMask’s unifying approach simplifies the pipeline for developers and practitioners by offering a versatile solution that does not require separate specialized models for tracking and segmentation tasks. Future developments could focus on optimizing the inference speed further, exploring lightweight architectures tailored for edge devices, and enhancing robustness against complex real-world scenarios like heavy occlusions or rapid object deformations.

Conclusion

In conclusion, the presented work provides a significant contribution to the fields of object tracking and video object segmentation. By leveraging the strengths of Siamese networks and introducing an effective multi-task learning framework, SiamMask not only achieves high accuracy but also ensures practical real-time applicability. This positions it as a strong candidate for both academic benchmarking and real-world deployment, warranting further exploration and optimization in subsequent research endeavors.