Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation

Abstract: We introduce a method for simultaneously classifying, segmenting and tracking object instances in a video sequence. Our method, named MaskProp, adapts the popular Mask R-CNN to video by adding a mask propagation branch that propagates frame-level object instance masks from each video frame to all the other frames in a video clip. This allows our system to predict clip-level instance tracks with respect to the object instances segmented in the middle frame of the clip. Clip-level instance tracks generated densely for each frame in the sequence are finally aggregated to produce video-level object instance segmentation and classification. Our experiments demonstrate that our clip-level instance segmentation makes our approach robust to motion blur and object occlusions in video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset, outperforming the ICCV 2019 video instance segmentation challenge winner despite being much simpler and using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).

An Expert Analysis of "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation"

The paper "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation" presents MaskProp, a novel method for video instance segmentation that effectively addresses the challenges of classifying, tracking, and segmenting object instances in video sequences. Developed by Gedas Bertasius and Lorenzo Torresani, MaskProp demonstrates significant accuracy in handling motion blur and occlusion with a reduced dataset compared to existing models, notably outperforming the ICCV 2019 video instance segmentation challenge winner on the YouTube-VIS dataset.

Methodology Overview

MaskProp introduces a mask propagation branch to adapt the Mask R-CNN architecture for video, allowing the model to generate clip-level instance tracks by propagating frame-level instance masks across video clips. This approach departs from the common practices of independent classification, detection, and tracking, offering a single model that performs all tasks end-to-end. By computing clip-level instance tracks centered on a middle frame, MaskProp aggregates these tracks densely over the video sequence, producing coherent video-level object instance segmentation and classification.

The core technical innovation, the mask propagation branch, facilitates feature propagation by predicting motion offsets between frames using deformable convolution. This enables MaskProp to handle disocclusions and occlusions effectively, providing robustness to nuisances like object instance overlapping and changes in pose.

Empirical Findings

Experiments conducted using the YouTube-VIS dataset reveal MaskProp's superior performance, achieving a mean average precision (mAP) of 46.6%, which significantly surpasses both the MaskTrack R-CNN and the EnsembleVIS method from the ICCV 2019 challenge. Notably, the latter relies on a far larger pre-training dataset. When MaskProp is further pre-trained on OpenImages, the mAP increases to 50.0%, underscoring the model's scalability and proficiency with additional training data.

Implications and Speculation

Practically, MaskProp's robust instance segmentation and tracking enhance real-world applications, such as automated surveillance, video annotation, and advanced video search capabilities. Theoretically, MaskProp signals a shift towards integrated, unified models that efficiently handle multi-task problems in computer vision, suggesting an emphasis on designing systems that address spatiotemporal consistencies holistically.

In terms of future developments, MaskProp indicates potential pathways in AI research for incorporating domain-specific adjustments, such as adapting to scenarios when only bounding boxes are available, or expanding to tasks like pose estimation. Such advancements could further streamline processing pipelines and reduce dependency on large-scale annotations.

Conclusion

MaskProp exemplifies a substantial step forward in video instance segmentation by offering a streamlined, data-efficient, and high-performing solution. The paper not only posits a technically competent model but also sets a precedent for future exploration into joint segmentation-tracking frameworks. As AI systems push boundaries in video processing, MaskProp may act as a foundational benchmark and inspire developments across related computer vision tasks.